A video segmentation method and system

Abstract

The application discloses a video segmentation method and system, and relates to the technical field of video segmentation. The method comprises the following steps: acquiring a target video to be segmented; extracting a multi-scale feature map of the target video to be segmented by using an image encoder; introducing key point coding, modeling motion features of a segmentation target, and forming shape constraints; estimating key points of un-labeled frames based on labeled true value information, time difference features of adjacent frames, and image feature maps, and forming coherent motion information; fusing image space features and motion features, completing time sequence segmentation under the guidance of the motion features, and obtaining a video segmentation image; and the method guides the segmentation process through key motion information, and improves the video segmentation precision.

Description

Technical Field

[0001] This invention relates to the field of video segmentation technology, and specifically to a video segmentation method and system. Background Technology

[0002] Video segmentation, by classifying each frame of a video at the pixel level, can identify and separate different objects and backgrounds, which is crucial for understanding video content. However, accuracy is not the only goal of video segmentation. Taking medical video segmentation as an example, accurately segmenting the target chamber or organ is only a basic requirement for clinical applications; the segmentation results should support further downstream tasks to expand its application attributes. For example, the purpose of segmenting the left ventricle in a cardiac ultrasound video is to identify the target chamber, perform clinical attribute measurements based on the segmentation results, and then assess cardiac function. In static image segmentation, because the target object is not dynamically changing, only spatial learning is needed to ensure segmentation accuracy. In dynamic image (video) segmentation, the introduction of the temporal dimension requires additional learning to cope with temporal changes. Therefore, video segmentation methods are usually designed to address the temporal segmentation difficulties brought about by the increased temporal dimension.

[0003] Existing video segmentation methods typically consider two approaches. The first is to extract features independently in the temporal dimension (unlike static images where learning is limited to the spatial dimension), then fuse the temporal and spatial information to obtain the final segmentation result. The second approach involves building a memory repository to store features from every frame of the video (or specific frames selected according to certain rules), and then designing a matching mechanism to match the most similar feature from the memory repository for the current frame to complete the segmentation. The challenges of video segmentation compared to static image segmentation lie firstly in the fact that the shape and position of the segmentation target dynamically change over time, making it difficult for segmentation methods to simultaneously learn both temporal and spatial information. Secondly, motion information, as a crucial component of temporal cues, is often undervalued in video segmentation methods, resulting in a lack of necessary shape constraints to guide the model to achieve temporally consistent segmentation results. Thirdly, segmentation learning heavily relies on manually labeled information; obtaining frame-by-frame annotations in video sequences is costly, and sparse annotations make it difficult for the model to learn enough features to ensure segmentation accuracy.

[0004] In summary, segmentation and motion estimation are two independent tasks. Static image segmentation does not involve motion estimation. Existing video segmentation methods either indirectly process motion information by learning through feature extraction in the time dimension without directly targeting motion information, or by segmenting through features of similar frames and fitting changes caused by motion using static information; or they are not targeted when using motion information: for example, they obtain motion-related vectors through optical flow calculation, but treat each part of the segmentation target indiscriminately, thus making it difficult to guarantee the segmentation accuracy of the video. Summary of the Invention

[0005] To address the shortcomings of existing technologies that rely on static information to fit changes caused by motion, making it difficult to guarantee the segmentation accuracy of videos, this invention proposes a video segmentation method and system. By introducing keypoint encoding, the motion features of the segmentation target are modeled to form shape constraints; based on labeled ground truth information and point encoding, unlabeled frames are estimated to form coherent motion information; and image spatial features and motion features are fused to complete temporal segmentation under the guidance of motion features, thereby solving the problems existing in the prior art.

[0006] A video segmentation method includes the following steps:

[0007] Obtain the video of the target object to be segmented, and extract the multi-scale feature map of each frame of the video to be segmented;

[0008] The process involves extracting boundary contour points from the mask annotations of the first frame of the video to be segmented, using these points as keypoints for the labeled frames; performing temporal interpolation on the keypoint positions of the labeled frames to obtain the initial positions of the keypoints in the next unlabeled frame; calculating the temporal difference features between the current unlabeled frame and its adjacent frames, fusing the temporal difference features with a multi-scale feature map matching its size, and then performing a convolution operation to obtain a point offset map; predicting the offset of the keypoints in the current unlabeled frame using the point offset map; and correcting the initial positions of the keypoints in the current unlabeled frame based on this offset to obtain the keypoint positions of the current unlabeled frame.

[0009] Based on the key point locations in the current frame, features are sampled from their corresponding multi-scale spatial feature maps to generate a key point feature matrix; the correlation between all point pairs in the key point feature matrix is ​​calculated to generate the embedding representation of the key points in the current frame; the embedding representation of the key points in the current frame is mapped to an attention map, and the attention map is multiplied point by point with the multi-scale feature map of the current frame to obtain the spatiotemporal fusion feature;

[0010] Decode the spatiotemporal fusion features to generate temporally consistent video segmentation images.

[0011] Furthermore, by inputting the video to be segmented into the ResNet deep learning feature extraction network, multi-scale feature maps are obtained. .

[0012] Furthermore, the step of sampling features from the corresponding multi-scale spatial feature map based on the key point positions of the current frame to generate a key point feature matrix specifically includes the following steps:

[0013] For time t Coordinates are The n Key points ,by Expand one at the center The support region of a pixel is used to form a support point matrix. ,but In the multi-scale feature map Corresponding position on the layer for:

[0014] ;

[0015] For the support area, the first j Line number k For each support point in a column, the multi-scale feature maps of its corresponding positions are concatenated along the dimension to obtain the feature map of each support point at each layer. c The dimensional eigenvector, then in the th dimension... i The keypoint feature matrix on the multi-scale feature map is then... ;

[0016] right L2 normalization is performed to obtain the key point feature matrix. :

[0017] ;

[0018] in, .

[0019] Furthermore, the correlation between all point pairs in the keypoint feature matrix is ​​calculated to generate the embedded representation of the keypoints in the current frame. The specific process includes the following steps:

[0020] Calculate the th keypoint feature matrix after normalization j Line number k Column and number l Line number m The correlation between different support points of a column is used to obtain a correlation matrix, which is represented as follows:

[0021] ;

[0022] The correlation matrix is ​​flattened using a two-layer MLP to obtain the embedded representation of the key points in the current frame:

[0023] .

[0024] Furthermore, the step of calculating the temporal difference features between the current unlabeled frame and its neighboring frames, fusing the temporal difference features with multi-scale feature maps that match their size, and then performing a convolution operation to obtain a point offset map specifically includes the following steps:

[0025] Video Convert to grayscale video And calculate the current frame and the time interval between them is tg Temporal difference features of adjacent frames :

[0026] ;

[0027] At every moment t ,Will and with Size-matched multi-scale features Channels are generated uniformly. and ;Will and Fusion, resulting in fusion characteristics :

[0028] ;

[0029] Fusion features By integrating convolution and activation functions, a point offset map is obtained. :

[0030] .

[0031] Furthermore, the process involves predicting the offset of key points in the current unlabeled frame using a point offset map; and then correcting the initial position of the key points in the current unlabeled frame based on this offset to obtain the position of the key points in the current unlabeled frame. Specifically, it means:

[0032] ;

[0033] in, These represent the key points of the marked frames.

[0034] Further, the process of mapping the embedded representation of the key points of the current frame to an attention map, and then multiplying the attention map point-by-point with the multi-scale feature map of the current frame to obtain the spatiotemporal fusion feature, specifically includes the following steps:

[0035] For each layer of feature map Embed the key points of the current frame into the representation. Mapped to attention map ;

[0036] attention map The spatiotemporal fusion features are obtained by multiplying them point-by-point with the multi-scale feature map of the current frame. :

[0037] ;

[0038] .

[0039] Furthermore, the spatiotemporal fusion features are decoded to generate temporally consistent video segmentation images; specifically, this is achieved by decoding the spatiotemporal fusion features... After performing bottom-up, layer-by-layer upsampling fusion, the following is adopted: Convolution generates segmentation masks This leads to temporally consistent video segmentation images:

[0040] ;

[0041] .

[0042] The present invention also includes a video segmentation system, comprising:

[0043] The acquisition module is used to acquire the video of the target object to be segmented and extract the multi-scale feature map of each frame of the video to be segmented.

[0044] The keypoint estimation module extracts boundary contour points from the mask annotations of the first frame of the video to be segmented as keypoints of the labeled frame; performs temporal interpolation on the keypoint positions of the labeled frames to obtain the initial positions of keypoints in the next unlabeled frame; calculates the temporal difference features between the current unlabeled frame and its adjacent frames, fuses the temporal difference features with a multi-scale feature map matching its size, and performs convolution operation to obtain a point offset map; predicts the offset of the keypoints in the current unlabeled frame using the point offset map; and corrects the initial positions of the keypoints in the current unlabeled frame based on the offset to obtain the keypoint positions of the current unlabeled frame.

[0045] The encoding module is used to sample features from the corresponding multi-scale spatial feature map based on the key point position of the current frame to generate a key point feature matrix; calculate the correlation between all point pairs in the key point feature matrix to generate the embedding representation of the key points of the current frame; map the embedding representation of the key points of the current frame to an attention map, and multiply the attention map with the multi-scale feature map of the current frame point by point to obtain the spatiotemporal fusion feature.

[0046] The decoding module is used to decode the spatiotemporal fusion features and generate temporally consistent video segmentation images.

[0047] This invention provides a video segmentation method with the following beneficial effects:

[0048] This invention introduces keypoint encoding into the segmentation process to model the motion features of the segmentation target, forming shape constraints. Based on labeled ground truth information, temporal difference features of adjacent frames, and image feature maps, keypoints of unlabeled frames are estimated to form coherent motion information. Finally, image spatial features and motion features are fused, and temporal segmentation is completed under the guidance of motion features. This method improves the segmentation effect of existing video segmentation methods. At the same time, because key motion information is used to guide the segmentation, the accuracy of video segmentation is improved, and advanced tasks such as measurement based on the segmentation results are supported. Attached Figure Description

[0049] Figure 1 This is a flowchart of the video segmentation method in an embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram comparing the results of the method of the present invention with those of existing segmentation methods in an embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of the measurement process in an embodiment of the present invention. Detailed Implementation

[0052] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0053] This invention proposes a video segmentation method, such as... Figure 1 As shown, the method specifically includes the following steps:

[0054] S1. Obtain video of any format from the video shooting device, and mark the segmentation mask of the target to be segmented on two of the video frames, which will be used as ground truth to guide model learning.

[0055] S2. Input the video frames into the image encoder (which can be any deep learning feature extraction network, such as ResNet) to obtain multi-scale image feature maps (spatial features).

[0056] Image encoder: Receives input video Multi-scale image feature maps are obtained through a feature extraction network. , Representing different scales.

[0057] S3, Key Point Estimation.

[0058] S3.1 Extract the boundary contour points as key points from the mask annotation of the first frame of the video.

[0059] S3.2 An alignment and fusion module is designed to estimate the keypoint locations in unlabeled frames under semi-supervised mode. This module extracts motion cues of the target by calculating the multi-frame temporal difference map of the grayscale video and combining it with the shallow image features with the highest spatial resolution among multi-scale features. Dynamic convolution is used to process the fused features to predict the offset of keypoints in each unlabeled frame. Based on this, keypoints in labeled frames are propagated to unlabeled frames through temporal interpolation and fine-tuned by combining the offset to obtain the keypoint estimation results for unlabeled frames, thereby establishing coherent motion information frame by frame in the video.

[0060] Alignment: First, align the video... Convert to grayscale video And calculate the current frame and the time interval between it. tg Temporal difference features of adjacent frames :

[0061] ;

[0062] At every moment t ,Will and with Size-matched multi-scale features The channels are formed in a unified manner. and Then merge them:

[0063] ;

[0064] Fusion features By integrating convolution and activation functions, a point offset map is obtained. :

[0065] ;

[0066] S3.3, Key points based on estimated frames Interpolation is performed to form the initial position of key points at the current unestimated time. The offset of key points in the current unlabeled frame is predicted by the point offset map. The initial position of key points in the current unlabeled frame is corrected according to the offset to obtain the position of key points in the current unlabeled frame, thereby establishing the coherent motion information frame by frame in the video.

[0067] S4. An innovative point encoder was designed to run in parallel with the image encoder; based on the point offset map. Calculated and estimated key point locations Afterwards, The data is fed into a point encoder to obtain point embeddings for unlabeled frames, thereby completing the motion information for each frame: .

[0068] Point encoding: First, extract the boundary contour points from the segmentation mask as key points. t Time coordinates are The n Key points To reduce interference from factors such as noise and point offset, Expand one at the center The receptive field of a pixel is used to form a support point matrix using points within that region. , In the multi-scale feature map Corresponding position on the layer for:

[0069] ;

[0070] For the support area, the first j Line number k For each support point in a column, the image feature maps of its corresponding positions are concatenated along the dimension to obtain the feature map of each support point in each layer. c The dimensional eigenvector, then in the th dimension... i The keypoint feature matrix on the multi-scale feature map is then... .right L2 normalization is performed to obtain :

[0071] ;

[0072] in To prevent the denominator from being 0.

[0073] Subsequently, regarding the first j Line number k Column and number l Line number m For different support points of a column, calculate their correlation to form a correlation matrix for the key points:

[0074] ;

[0075] The correlation matrix is ​​flattened using a two-layer MLP to obtain a point embedding representation to form local spatial constraints:

[0076] .

[0077] This invention introduces point coding during the segmentation process to establish motion information for guiding temporal segmentation; based on the designed point coding, coherent motion information is formed by estimating key points frame by frame.

[0078] S5. The key points are embedded and mapped into an attention map and multiplied point by point with the image features to highlight key motion information. The feature distribution is smoothed by convolution to obtain a multi-scale feature representation that integrates motion and spatial information.

[0079] Feature fusion: For each layer of feature map Using keypoint embedding Generate attention map The spatiotemporal fusion features are obtained by weighting and enhancing the image features (i.e., spatial features) generated by the image encoder. :

[0080] ;

[0081] .

[0082] This invention forms spatiotemporal fusion features by fusing image features and motion information.

[0083] S6. Decode the spatiotemporal fusion features: Upsample and fuse the spatiotemporal fusion features layer by layer to gradually restore the spatial resolution. Finally, generate a segmentation mask through convolution and activation functions to complete the temporally consistent video segmentation.

[0084] Mask decoding: Feature maps aggregated and fused by the decoder It employs a bottom-up, layer-by-layer upsampling fusion method, ultimately achieving... Convolution generates segmentation masks :

[0085] ;

[0086] .

[0087] This invention improves the segmentation effect of existing video segmentation methods; at the same time, because it uses key motion information to guide the segmentation process, it supports advanced tasks such as measurement based on the segmentation results.

[0088] Experiments have shown that: Figure 2 Taking cardiac ultrasound video as an example, segmentation probability graphs of several different segmentation methods are shown. Darker colors in the segmentation probability graphs represent higher reliability. Results (a) are from MemSAM, a method designed for cardiac ultrasound video segmentation; results (b) are from MedSAM, a method optimized for medical image segmentation; results (c) are from H2Former, a method designed for medical image segmentation; (d) and (e) are XMem and iMOS, methods designed for video segmentation, respectively; and (f) is the method of this invention. Figure 2As shown, other segmentation methods exhibit a dark color at the center of the segmented target and a light color at the boundary in the segmentation probability map. This is because the certainty of the segmentation method for determining the target decreases at the boundary. However, the method proposed in this invention, due to motion information constraints, results in a segmentation result that is almost entirely dark, with only some points at the boundary showing a light color.

[0089] Furthermore, measurements can be performed based on the method of this invention. Taking the opening diameter and depth of the left atrial appendage as an example, the measurement results are as follows: Figure 3 As shown.

[0090] Based on the same inventive concept, this invention also proposes a video segmentation system, comprising:

[0091] The acquisition module is used to acquire the video of the target object to be segmented and extract the multi-scale feature map of each frame of the video to be segmented.

[0092] The keypoint estimation module extracts boundary contour points from the mask annotation of the first frame of the video to be segmented as keypoints of the labeled frame; performs temporal interpolation on the keypoint positions of the labeled frames to obtain the initial positions of keypoints in the next unlabeled frame; calculates the temporal difference features between the current unlabeled frame and its adjacent frames, fuses the temporal difference features with a multi-scale feature map matching its size, and performs convolution operation to obtain a point offset map; predicts the offset of the keypoints of the current unlabeled frame using the point offset map; and corrects the initial positions of the keypoints of the current unlabeled frame based on the offset to obtain the keypoint positions of the current unlabeled frame.

[0093] The encoding module is used to sample features from the corresponding multi-scale spatial feature map based on the key point position of the current frame to generate a key point feature matrix; calculate the correlation between all point pairs in the key point feature matrix to generate the embedding representation of the key points of the current frame; map the embedding representation of the key points of the current frame to an attention map, and multiply the attention map with the multi-scale feature map of the current frame point by point to obtain the spatiotemporal fusion feature.

[0094] The decoding module is used to decode the spatiotemporal fusion features and generate temporally consistent video segmentation images.

[0095] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A video segmentation method, characterized in that, Includes the following steps: Obtain the video of the target object to be segmented, and extract the multi-scale feature map of each frame of the video to be segmented; Extract boundary contour points from the mask annotation of the first frame of the video to be segmented as key points of the labeled frames; Temporal interpolation is performed on the keypoint positions of the labeled frames to obtain the initial positions of the keypoints in the next unlabeled frame; the temporal difference features between the current unlabeled frame and its adjacent frames are calculated, and the temporal difference features are fused with a multi-scale feature map that matches their size and then convolved to obtain a point offset map; the offset of the keypoints in the current unlabeled frame is predicted using the point offset map; the initial positions of the keypoints in the current unlabeled frame are corrected based on the offset, and the keypoint positions in the current unlabeled frame are obtained; Based on the key point locations in the current frame, features are sampled from the corresponding multi-scale spatial feature map to generate a key point feature matrix. Calculate the correlation between all point pairs in the keypoint feature matrix to generate the embedding representation of the keypoints in the current frame; map the embedding representation of the keypoints in the current frame to an attention map, and multiply the attention map point by point with the multi-scale feature map of the current frame to obtain the spatiotemporal fusion feature; Decode the spatiotemporal fusion features to generate temporally consistent video segmentation images.

2. The video segmentation method according to claim 1, characterized in that, By inputting the video to be segmented into the ResNet deep learning feature extraction network, multi-scale feature maps are obtained. .

3. The video segmentation method according to claim 1, characterized in that, The key point feature matrix is ​​generated by sampling features from the corresponding multi-scale spatial feature map based on the key point positions of the current frame. Specifically, the following steps are included: For time t Coordinates are The n Key points ,by Expand one at the center The support region of a pixel is used to form a support point matrix. ,but In the multi-scale feature map Corresponding position on the layer for: ； For the support area, the first j Line number k For each support point in a column, the multi-scale feature maps of its corresponding positions are concatenated along the dimension to obtain the feature map of each support point at each layer. c The dimensional eigenvector, then in the th dimension... i The keypoint feature matrix on the multi-scale feature map is then... ; right L2 normalization is performed to obtain the key point feature matrix. : ； in, ; ; Used to prevent the denominator from being 0.

4. The video segmentation method according to claim 3, characterized in that, The process of calculating the correlation between all point pairs in the keypoint feature matrix and generating the embedded representation of the keypoints in the current frame includes the following steps: Calculate the th keypoint feature matrix after normalization j Line number k Column and number l Line number m The correlation between different support points of a column is used to obtain a correlation matrix, which is represented as follows: ； The correlation matrix is ​​flattened using a two-layer MLP to obtain the embedded representation of the key points in the current frame: 。 5. The video segmentation method according to claim 1, characterized in that, The process involves calculating the temporal difference features between the currently unlabeled frame and its neighboring frames, fusing these temporal difference features with a multi-scale feature map that matches their size, and then performing a convolution operation to obtain a point offset map. Specifically... Includes the following steps: Video Convert to grayscale video And calculate the current frame and the time interval between them is tg Temporal difference features of adjacent frames : ； At every moment t ,Will and with Size-matched multi-scale features Channels are generated uniformly. and ;Will and Fusion, resulting in fusion characteristics : ； Fusion features By integrating convolution and activation functions, a point offset map is obtained. : 。 6. The video segmentation method according to claim 5, characterized in that, The process involves predicting the offset of key points in the current unlabeled frame using a point offset map; then correcting the initial position of the key points in the current unlabeled frame based on this offset to obtain the position of the key points in the current unlabeled frame. Specifically, it means: ； in, These represent the key points of the marked frames.

7. The video segmentation method according to claim 4, characterized in that, The process of mapping the embedded representation of key points in the current frame to an attention map, and then multiplying the attention map point-by-point with the multi-scale feature map of the current frame to obtain spatiotemporal fusion features, specifically includes the following steps: For each layer of feature map Embed the key points of the current frame into the representation. Mapped to attention map ; attention map The spatiotemporal fusion features are obtained by multiplying them point-by-point with the multi-scale feature map of the current frame. : ； 。 8. The video segmentation method according to claim 7, characterized in that, The process involves decoding the spatiotemporal fusion features to generate temporally consistent video segmentation images; specifically, this is achieved through decoding the spatiotemporal fusion features... After performing bottom-up, layer-by-layer upsampling fusion, the following is adopted: Convolution generates segmentation masks This leads to temporally consistent video segmentation images: ； 。 9. A video segmentation system, characterized in that, include: The acquisition module is used to acquire the video to be segmented of the target object and extract the multi-scale feature map of each frame of the video to be segmented. The key point estimation module is used to extract boundary contour points from the mask annotation of the first frame of the video to be segmented as key points of the labeled frame; Temporal interpolation is performed on the keypoint positions of the labeled frames to obtain the initial positions of the keypoints in the next unlabeled frame; the temporal difference features between the current unlabeled frame and its adjacent frames are calculated, and the temporal difference features are fused with a multi-scale feature map that matches their size and then convolved to obtain a point offset map; the offset of the keypoints in the current unlabeled frame is predicted using the point offset map; the initial positions of the keypoints in the current unlabeled frame are corrected based on the offset, and the keypoint positions in the current unlabeled frame are obtained; The encoding module is used to sample features from the corresponding multi-scale spatial feature map based on the key point positions of the current frame and generate a key point feature matrix. Calculate the correlation between all point pairs in the keypoint feature matrix to generate the embedding representation of the keypoints in the current frame; map the embedding representation of the keypoints in the current frame to an attention map, and multiply the attention map point by point with the multi-scale feature map of the current frame to obtain the spatiotemporal fusion feature; The decoding module is used to decode the spatiotemporal fusion features and generate temporally consistent video segmentation images.

Images