A semi-supervised video object segmentation method and system with mask correction

Abstract

The application discloses a kind of semi-supervised video target segmentation method and system of mask correction.The implementation scheme is as follows, first, video sequence dataset and segmentation label are acquired, and data set video sequence is expanded and handled, second, segmentation model is constructed, and loss function is constructed to train segmentation model, finally, the trained segmentation model is used to carry out video target segmentation.The application captures the dependent relationship between the structure of feature space by designing detail perception decoder module, adaptively selects the spatial information that needs to be focused on, focuses on the target channel of feature to enhance the detailed information of feature, and removes the information unrelated to the target in feature, second, design mask correction module, estimate the uncertain area of mask according to rough prediction result, correct the target boundary and other uncertain areas using local detailed features, which can effectively improve the segmentation accuracy of model and solve the problem of cumulative segmentation error.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more specifically, to a semi-supervised video target segmentation method and system with mask correction. Background Technology

[0002] The main task of video object segmentation is to continuously segment the target object of interest from a video sequence. This paper focuses on semi-supervised video object segmentation, where the target instance is defined by a given accurate segmentation mask in the first frame, and the segmentation result of the target instance is predicted in the remaining video frames. Video object segmentation is one of the most challenging tasks in computer vision, with wide-ranging applications in interactive video editing, augmented reality, and autonomous driving.

[0003] In recent years, with the rapid development of deep learning and convolutional networks, more and more video object segmentation methods based on convolutional networks have emerged. Researchers prefer video object segmentation methods based on spatiotemporal memory networks, which not only have an advantage in segmentation speed but also achieve good accuracy. In 2019, Seoung Wug Oh et al. first proposed spatiotemporal memory networks (Seoung Wug Oh, Joon-Young Lee, Ning Xu, Seon Joo Kim. Video objectsegmentation using space-time memory networks. ICCV 2019:9225-9234.). This method uses the current video frame to be segmented as the query frame and the previously segmented video frames and their segmentation masks as memory frames, obtaining pixel-level memory content by performing dense matching between the two. Based on the idea of ​​space-time memory networks, Ho Kei Cheng et al. proposed the STCN method (Ho Kei Cheng, Yu-Wing Tai, Chi-Keung Tang. Rethinking space-time networks with improved memory coverage for efficient video objectsegmentation. NeurIPS 2021:11781-11794). Unlike space-time memory networks, this method directly establishes correspondences between frames without re-encoding the mask features of each object, thus forming an efficient and robust framework.

[0004] However, accurate segmentation of target boundaries is also a key factor affecting the performance of semi-supervised video target segmentation. Spatiotemporal memory network-based methods are only suitable for single-target segmentation. Therefore, in multi-target scenarios, the video target segmentation model must match and segment each target separately, ignoring the competition between different targets in multi-target segmentation, leading to blurred target boundaries. Furthermore, spatiotemporal memory networks use encoder-decoder segmentation masks, and upsampling low-resolution target masks results in blurred target boundaries and poor-quality segmentation mask results. When this erroneous segmentation information is used as memory frames to guide the segmentation of the current video frame, it leads to severe error accumulation, affecting the accurate segmentation of the current video frame. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a semi-supervised video target segmentation method and system with mask correction. The method accepts a memory frame sequence and the current frame as input. Feature information is extracted through a spatiotemporal memory network encoder, including a Key encoder and a Value encoder. A spatiotemporal memory reading module obtains the Value feature of the current query frame. A detail-aware decoder is developed, which reconstructs the decoder's input features by fusing deep features from the Key encoder, establishing a correlation between global and local features to obtain detail-enhanced decoding features. A coarse prediction mask is obtained through progressive upsampling. A mask correction module is developed, which estimates the uncertain regions of the mask based on the coarse prediction results and corrects these uncertain regions using a mask optimization mechanism. Finally, the entire network is optimized by minimizing the cross-entropy segmentation loss and the uncertainty region confidence loss.

[0006] To achieve the above objectives, the present invention provides a semi-supervised video target segmentation method and system with mask correction, comprising the following steps;

[0007] S1. Obtain the dataset and segmentation labels:

[0008] Obtain the video object segmentation dataset and the static image dataset, as well as the segmentation labels corresponding to these two datasets. Combine each image in the dataset with its corresponding segmentation label into an image pair.

[0009] S2. Data expansion and processing, specifically including the following steps:

[0010] S21. After normalizing each image pair consisting of the static image dataset and corresponding segmentation labels obtained in step S1, repeat the following process to obtain the synthetic video training samples corresponding to each image pair. The set of synthetic video training samples constitutes the synthetic video training set:

[0011] S211. Shrink the short side of the image pair to w pixels, and shrink the long side proportionally to the short side. Randomly crop the resulting image pair to h×w pixels, where w is the width of the cropped image, h is the height of the cropped image, and w and h are both positive integers.

[0012] S212. Apply random scaling, random horizontal flipping, random color jittering and random grayscale conversion to the cropped image pair in sequence to obtain the enhanced image pair corresponding to the image pair.

[0013] S213. Repeat step S212 three times to obtain three enhanced image pairs corresponding to the image pair. These three enhanced image pairs form a synthetic video training sample.

[0014] S22. After normalizing each video and its corresponding segmentation label in the video target segmentation dataset obtained in step S1, repeat the following process to obtain the real video training samples corresponding to each video. The set of real video training samples constitutes the real video training set:

[0015] S221. Randomly select three image pairs from the video and its corresponding segmentation labels;

[0016] S222. Reduce the short side of the three image pairs to w pixels, and reduce the long side proportionally. Then, randomly crop the three image pairs to h×w pixels. The meaning and value of w and h are the same as in step S21.

[0017] S223. Apply random cropping, color jittering and random grayscale conversion to these three image pairs in sequence to obtain three enhanced image pairs. These three enhanced image pairs form a real video training sample.

[0018] S3. Construct a segmentation model, which consists of a memory pool, an encoder, a global context-aware module, a spatiotemporal memory retrieval module, a detail-aware decoder, and a mask correction module. The specific steps include:

[0019] S31. Construct a memory pool by placing the 1st frame, τ+1st frame, 2τ+1st frame, ..., Nτ+1st frame of the video sequence and their corresponding segmentation tags into the memory pool, where τ is a positive integer with a value range of [1, 100]. τ C The relative position of the current frame, symbol Indicates rounding down;

[0020] S32. Construct a key encoder using a convolutional neural network. The memory frame and the current frame sequentially pass through the first four layers of this encoder. The outputs of the first layer are fq1 and fm1, the second layer is fq2 and fm2, the third layer is fq3 and fm3, and the fourth layer is fq4 and fm4. Then, fq4 and fm4 are processed using mapping convolutions to finally obtain K. Q and K M ;

[0021] S33. Construct a Value encoder, which consists of a convolutional neural network and a feature multiplexer. The feature multiplexer consists of one CBAM block and two residual blocks. All memory frames in the memory pool and their segmentation masks are processed through the first four layers of the convolutional neural network to obtain the fourth layer output fv4. fv4 is concatenated with fm4, the output of the Key encoder, and then processed by the feature multiplexer to obtain the final Value feature output V. M ;

[0022] S34. Construct a spatiotemporal memory retrieval module to calculate the pairwise similarity between the current video frame to be segmented and the video frames in the memory pool using a non-local attention approach, and obtain the spatiotemporal information V of the current frame's Value feature. Q ;

[0023] S35. Construct a Detail-Aware Decoding (DAD). The DAD consists of a feature input processing part and a feature output processing part. The feature input processing part consists of one CBAM block and two residual modules. The residual modules consist of three convolutional layers and two ReLU activation layers. The feature output processing part consists of two upsampling modules and one prediction convolutional module. The first upsampling module consists of four convolutional layers, two ReLU activation layers and one bilinear interpolation. The second upsampling module consists of three convolutional layers, two ReLU activation layers and one bilinear interpolation. The prediction convolutional module consists of one convolutional layer, one ReLU activation layer and one bilinear interpolation.

[0024] fq4 obtained in step S32 and V obtained in step S34 Q After concatenation, the input is fed into the feature input processing part of the detail-aware decoder to obtain f01; f01 and fq3 obtained in step S32 are fed into the first upsampling module of the detail-aware decoder to obtain f02; f02 and fq2 obtained in step S32 are fed into the second upsampling module of the detail-aware decoder to obtain f03; f03 is fed into the prediction convolution module of the detail-aware decoder to obtain the coarse predicted segmentation label M. init ;

[0025] S36. Construct a Mask Correction Module (MCM), which consists of an uncertain region estimation module, an average pooling layer, a max pooling layer, a mapping convolutional layer, a residual module, and a prediction convolutional layer.

[0026] S361. To measure the classification uncertainty of each pixel, the coarse probability map M output from step S35 is used. init Calculate the corresponding uncertainty region estimation map U, and the result will be used as a confidence level to measure whether the pixel segmentation is reliable.

[0027] S362. Obtain the rough probability map M from step S35. init Following step S32 to obtain fq1, for target i, M is first processed using average pooling layer and max pooling layer. init Together with fq1, we calculate the local reference feature y with target prior at any pixel (feature) location p. i (p):

[0028]

[0029] Similarly, for each coarse probability map of the target and background, the same operation can be performed on each pixel to obtain the local reference feature map y.

[0030] S363. Compare the similarity between the local feature fq1 obtained in step S32 and the local reference feature y obtained in step S362. That is, after concatenating y and fq1, process them through a mapping convolutional layer and then use a residual network module to learn and predict the local feature similarity to obtain the feature map f(y, fq1). q1 ):

[0031] f(r1,y)=Conv2(Resblock(Conv1(Concat(r1,y)))),

[0032] S364. Using the max pooling layer combined with the maximum probability value in the neighborhood of pixel p, the probability map m of local optimization is calculated as follows:

[0033]

[0034] m = Score * f(r1, y),

[0035] S365. Add the local optimization probability map m obtained in step S364 to the coarse probability map M obtained in step S35. init Simultaneously, the uncertainty probability map U obtained in step S361 is used to weight the local optimization probability map m obtained in step S364, making it more focused on the correction of uncertain regions; thus, the final accurate video target segmentation prediction M is obtained as follows:

[0036] M = M init +U*m,

[0037] Finally, the segmentation result M is bilinearly upsampled by a factor of 2 to obtain the network output mask result image, thus completing the construction of the segmentation model;

[0038] S4. Construct the loss function, where the total loss function L total Using the cross-entropy loss function L and the uncertainty region confidence loss function L conf The sum is defined as follows:

[0039] L total =α*L+β*L conf ,

[0040] S41. Construct the cross-entropy loss function L, defined as follows:

[0041]

[0042] Where Y is the true label. It predicts segmentation labels, H Y and W Y These are the height and width of the actual segmentation label, Y. ij It is the pixel value in the i-th row and j-th column of Y. yes The pixel value in the i-th row and j-th column, where i = 1, 2, ..., H Y j = 1, 2, ..., W Y log(.) represents the natural logarithm;

[0043] S42. Construct the confidence loss function for the uncertain region, defined as follows:

[0044] L conf =||U||2,

[0045] Where U is the probability map of the uncertain region obtained in step S371, and ||.||2 represents the Euclidean norm;

[0046] S5. Train the segmentation model. Use the static image-to-video training set obtained in S21 to train the segmentation model constructed in S3. Obtain the loss value according to the loss function constructed in S4, and use stochastic gradient descent to update the parameters in the segmentation model to obtain the pre-trained model. Use the real video training set obtained in S22 to train the pre-trained model. Obtain the loss value according to the loss function constructed in S4, and use stochastic gradient descent to update the parameters in the model to obtain the trained segmentation model.

[0047] S6. Video target segmentation: Obtain the video to be segmented and the segmentation label corresponding to its first frame. Starting from the second frame of the video to be segmented, input them sequentially into the segmentation model trained in S5 and output the segmentation label.

[0048] Compared with the prior art, the present invention has the following advantages:

[0049] (1) The detail-aware decoder module proposed in this invention adaptively selects the spatial information to be focused on by capturing the dependency relationship between feature spatial structures, and enhances the detail information of the feature by focusing on the target channel of the feature, and removes information in the feature that is not related to the target.

[0050] (2) The mask correction module proposed in this invention estimates the uncertain region of the mask based on the rough prediction result and corrects the target boundary and other uncertain regions using local detail features. This can effectively improve the model segmentation accuracy and solve the challenging problem of segmentation error accumulation. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 Flowchart of the method in this invention patent;

[0053] Figure 2 The segmentation model structure diagram in this invention patent;

[0054] Figure 3 Detail-aware decoder input processing structure diagram in this invention patent;

[0055] Figure 4 Structure diagram of the mask correction module in this invention patent;

[0056] Figure 5 This invention patent provides a comparison of target boundary segmentation results in challenging scenarios with those of other methods. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other. The invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0058] The specific implementation methods of this invention patent are described below;

[0059] Figure 1 The diagram shows a semi-supervised video target segmentation method and system flowchart with mask correction according to this invention patent. The specific process is as follows:

[0060] Step S1, Obtain the dataset and segmentation labels:

[0061] Obtain the video object segmentation dataset and the static image dataset, along with their corresponding segmentation labels. Combine each image in the dataset with its corresponding segmentation label to form an image pair.

[0062] Step S2, data augmentation and processing, specifically includes the following steps:

[0063] S21. After normalizing each image pair consisting of the static image dataset and corresponding segmentation labels obtained in step S1, repeat the following process to obtain the synthetic video training samples corresponding to each image pair. The set of synthetic video training samples constitutes the synthetic video training set:

[0064] S211. Shrink the short side of the image pair to w pixels, and shrink the long side proportionally to the short side. Randomly crop the resulting image pair to h×w pixels, where w is the width of the cropped image, h is the height of the cropped image, and w and h are both positive integers.

[0065] S212. Apply random scaling, random horizontal flipping, random color jittering and random grayscale conversion to the cropped image pair in sequence to obtain the enhanced image pair corresponding to the image pair.

[0066] S213. Repeat process II three times to obtain three enhanced image pairs corresponding to the image pair. These three enhanced image pairs form a synthetic video training sample.

[0067] S22. After normalizing each video and its corresponding segmentation label in the video target segmentation dataset obtained in step S1, repeat the following process to obtain the real video training samples corresponding to each video. The set of real video training samples constitutes the real video training set:

[0068] S221. Randomly select three image pairs from the video and its corresponding segmentation labels;

[0069] S222. Reduce the short side of the three image pairs to w pixels, and reduce the long side proportionally. Then, randomly crop the three image pairs to h×w pixels. The meaning and value of w and h are the same as in step S21.

[0070] S223. Apply random cropping, color jittering, and random grayscale conversion to these three image pairs in sequence to obtain three enhanced image pairs. These three enhanced image pairs form a real video training sample.

[0071] Step S3: Construct the segmentation model. This segmentation model consists of an encoder, a memory pool, a global context-aware module, a spatiotemporal memory retrieval module, a detail-aware decoder, and a mask correction module. Figure 2 The diagram shown is a structural diagram of the segmentation model in this invention patent. The specific steps are as follows:

[0072] S31. Construct a memory pool by placing the 1st frame, τ+1st frame, 2τ+1st frame, ..., Nτ+1st frame of the video sequence and their corresponding segmentation labels into the memory pool, where τ is a positive integer with a value range of [1, 100]. τ C The relative position of the current frame, symbol This indicates rounding down; in this example, τ is set to 5.

[0073] S32. Using a ResNet50 convolutional neural network as the key encoder, the current frame and the memory frame sequentially pass through the first four layers of this encoder. The outputs of the first layer are fq1 and fm1, the second layer is fq2 and fm2, the third layer is fq3 and fm3, and the fourth layer is fq4 and fm4. Then, fq4 and fm4 are processed by mapping convolution to compress the number of feature channels to 512, thus obtaining K. Q and K M ;

[0074] S33. Construct the Value encoder, which consists of a ResNet18 convolutional neural network and a feature multiplexer. The feature multiplexer consists of one CBAM block and two residual blocks. All memory frames and their segmentation masks in the memory pool are processed through the first four layers of the ResNet18 convolutional neural network to obtain the fourth layer output fv4. fv4 is concatenated with fm4 obtained in step S32 to obtain fv. fv is first processed by a residual block, reducing the number of feature channels from 1280 to 512; then, it is enhanced by a CBAM block to obtain the Value encoder output V. M ;

[0075] S34. For the spatiotemporal memory reading module, the query key characteristics are obtained from step S34. Key features of memory And obtain the memory value feature in step S23 First, calculate K′ using the L2 distance. Q and K′ M The similarity matrix between them is normalized; then, V M Perform matrix multiplication and weighted summation to obtain the Value feature of the current query frame. The specific calculation process is as follows:

[0076]

[0077]

[0078] Where i and j are the spatial location indices for querying the Key feature and remembering the Key feature, respectively. It is a normalization factor. Represents the L2 distance metric function;

[0079] S35. Construct a Detail-Aware Decoding (DAD). The DAD consists of a feature input processing part and a feature output processing part. Figure 3 The feature input processing section consists of a CBAM block and two residual modules. The first residual module consists of three convolutional layers with 512 kernels and a kernel size of 3, and two ReLU activation layers. The second residual module consists of two convolutional layers with 512 kernels and a kernel size of 3, and two ReLU activation layers. The feature output processing section consists of two upsampling modules and one prediction convolutional module. The first upsampling module consists of one convolutional layer with 512 kernels and a kernel size of 3, three convolutional layers with 256 kernels and a kernel size of 3, two ReLU activation layers, and a bilinear interpolation module with a scaling factor of 2. The second upsampling module consists of three convolutional layers with 256 kernels and a kernel size of 3, two ReLU activation layers, and a bilinear interpolation module with a scaling factor of 2. The prediction convolutional module consists of one convolutional layer with 1 kernel and a kernel size of 3, one ReLU activation layer, and a bilinear interpolation module with a scaling factor of 2.

[0080] fq4 obtained in step S32 and V obtained in step S34 Q Cascade V is processed by the first residual block, and then V... Q Enhance the details and remove V. QThe data is not related to the target, and the number of feature channels is reduced from 1536 to 512. Then, the CBAM block captures the dependencies between the feature space structures, adaptively selects the spatial information that needs attention, focuses more on the target channel, and suppresses the background channel to obtain enhanced features. Finally, after further learning by the second residual block, f01 is obtained.

[0081] f 01 =Resblock2(Resblock1(V)+CBAM(Resblock1(V))),

[0082] Among them, Resblock i It is the i-th residual block, and CBAM(·) is the spatiotemporal attention feature enhancement process;

[0083] f01 and fq3 obtained in step S32 are input into the first upsampling module of the detail-aware decoder to obtain f02; f02 and fq2 obtained in step S32 are input into the second upsampling module of the detail-aware decoder to obtain f03; f03 is input into the prediction convolution module of the detail-aware decoder to obtain the coarse prediction segmentation label M. init ;

[0084] S36. Construct a mask correction module. Figure 4 The mask correction module shown consists of an uncertain region estimation module, an average pooling layer with a window size of 7, a max pooling layer with a window size of 7, a convolutional layer with 32 kernels and a kernel size of 3, a residual module, and a prediction convolutional layer. The residual module consists of two convolutional layers with 32 kernels and a kernel size of 3 and two ReLU activation layers. The prediction convolutional layer consists of a convolutional layer with 1 kernel and a kernel size of 3 and a ReLU activation layer.

[0085] S361. To measure the classification uncertainty of each pixel, the coarse probability map output from step S35 is used. Calculate the corresponding uncertainty region estimation map.

[0086]

[0087] Where K represents the total number of targets and background elements; It is M init The maximum value of each pixel in the K-dimensional space. It is M init The second largest value of each pixel in the K-dimensional region, and Since each pixel in the segmentation probability map can only belong to one target category, meaning the probability of a pixel belonging to a certain target category is much greater than that of other categories, this invention calculates the ratio of the maximum probability to the second-highest probability of a pixel on the target dimension K, and normalizes it to between 0 and 1 using an exponential function; the result will be used as a confidence level to measure whether the pixel segmentation is reliable; U∈(0,1], the smaller the value, the higher the confidence level of the pixel classification; conversely, it indicates that the pixel segmentation estimate is unreliable and will be divided into an uncertain segmentation region;

[0088] S362. Obtain the rough probability map M from step S35. init The process follows with step S32 to obtain fq1, where fq1 possesses high-fine-grained information (such as color, texture, edges, etc.) and has a scale of the same size as the coarse probability map, thus making it more suitable for correcting the segmentation prediction mask for uncertain regions such as target boundaries; for target i, M is first processed using an average pooling layer and a max pooling layer. init Together with fq1, we calculate the local reference feature with target prior at any pixel (feature) location p.

[0089]

[0090] in, Let q be a 7×7 neighborhood centered at pixel p. Any pixel in Let f be the probability value that pixel q belongs to target i. q1 (q) represents the local feature of pixel q; similarly, for each coarse probability map of the target and background, performing the same operation on each pixel yields the local reference feature map.

[0091] S363. Compare the similarity between the local feature fq1 obtained in step S32 and the local reference feature y obtained in step S362. That is, after concatenating y and fq1, process them through a mapping convolutional layer and then use a residual network module to learn and predict the local feature similarity to obtain the feature map f(y, fq1). q1 ):

[0092] f(y,f q1 )=Conv2(Resblock(Conv1(Concat(y,f q1 ))))

[0093] Wherein, Concat(.,.) is concatenated along the channel dimension, Resblock(·) is the residual module of the mask correction module, and Conv1(·) and Conv2(·) are the mapping convolutional layer and the prediction convolutional layer of the mask correction module, respectively.

[0094] S364. Using the max pooling layer combined with the maximum probability value in the neighborhood of pixel p, the probability map m of local optimization is calculated as follows:

[0095]

[0096] m = Score * f(r1, y),

[0097] Where Score(p) represents the maximum probability value in the neighborhood of pixel p calculated using the max pooling layer of the mask correction module. Similarly, the same operation is repeated for each pixel to obtain... This value can guide the local optimization mask towards a target with high confidence.

[0098] S365. Add the local optimization probability map m obtained in step S364 to the coarse probability map M obtained in step S35. init Simultaneously, the uncertainty probability map U obtained in step S361 is used to weight the local optimization probability map m obtained in step S364, making it more focused on the correction of uncertain regions; thus, the final accurate video target segmentation prediction M is obtained as follows:

[0099] M = M init +U*m,

[0100] Finally, the segmentation result M is bilinearly upsampled by a factor of 2 to obtain the network output mask result image, thus completing the construction of the segmentation model.

[0101] Step S4, construct the loss function, where the total loss function L total Using the cross-entropy loss function L and the uncertainty region confidence loss function L conf The sum is defined as follows:

[0102] L total =α*L+β*L conf ,

[0103] In this example, α is set to 1 and β to 0.5;

[0104] S41. Construct the cross-entropy loss function L, defined as follows:

[0105]

[0106] Where Y is the true label. It predicts segmentation labels, H Y and W Y These are the height and width of the actual segmentation label, Y. ij It is the pixel value in the i-th row and j-th column of Y. yes The pixel value in the i-th row and j-th column, where i = 1, 2, ..., HY j = 1, 2, ..., W Y log(.) represents the natural logarithm;

[0107] S42. Construct the confidence loss function for the uncertain region, defined as follows:

[0108] L conf =||U||2,

[0109] Where U is the probability map of the uncertain region obtained in step S371, and ||.||2 represents the Euclidean norm.

[0110] Step S5: Train the segmentation model; use the static image-to-video training set obtained in S21 to train the segmentation model constructed in S3, obtain the loss value according to the loss function constructed in S4, and use stochastic gradient descent to update the parameters in the segmentation model to obtain the pre-trained model; use the real video training set obtained in S22 to train the pre-trained model, obtain the loss value according to the loss function constructed in S4, and use stochastic gradient descent to update the parameters in the model to obtain the trained segmentation model.

[0111] Step S6, video target segmentation; obtain the video to be segmented and the segmentation label corresponding to its first frame, and input them sequentially into the segmentation model trained in step S5 starting from the second frame of the video to be segmented, and output the segmentation label M.

[0112] According to another aspect of this application, a semi-supervised video object segmentation system with mask correction is also provided, comprising the following units:

[0113] Video sequence input unit: Given a set of video frames, expand and process them according to the form in step S2 to obtain the input video sequence;

[0114] The model training unit, used to train a mask-corrected semi-supervised video object segmenter, comprises a memory pool module, a feature encoder module, a global context awareness module, a spatiotemporal memory information retrieval module, a segmentation decoder module, and a mask correction module. The memory pool module randomly selects three frames at certain intervals from the video sequence frame order as training video frames. The first frame contains the segmentation mask, the second frame onwards serves as the query frames to be segmented, and the video sequence before the query frames serves as memory frames. The feature encoder sequentially extracts the memory key features, query key features, and memory value features from the video frame images, and then stores the memory key features in the memory key. The query key feature, the memory value feature, and the memory value feature are input into the spatiotemporal memory information reading module. Based on the memory frame, the key feature information of the query frame is read to obtain a more discriminative query value feature. The query frame value feature is input into the detail-aware decoder to obtain a coarse mask. Based on the coarse mask, an uncertainty region probability map is obtained. Then, the uncertainty probability map, the coarse mask map, etc. are input into the mask correction module to obtain the corrected segmentation prediction mask map of the query frame. Finally, the target segmentation is trained by minimizing the cross-entropy loss between the prediction mask map and the real mask map and the uncertainty region confidence loss.

[0115] In the testing phase, the video target segmentation unit selects one frame from every five frames of the video sequence as described in step S31 and adds it to the memory pool as a memory frame. Starting from the second frame of the video to be segmented, it is sequentially input into the segmentation model trained in step S5 to perform video target segmentation for subsequent frames.

[0116] This system is used to implement the functions of the methods in the above embodiments. The specific implementation steps of the methods involved in the system module have been described in the methods and will not be repeated here.

[0117] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural transformations made using the contents of the present invention's specification and drawings under the inventive concept of the present invention, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. A semi-supervised video target segmentation method with mask correction, characterized in that, The method is executed by a computer and includes the following steps: Step S1: Obtain the dataset and segmentation labels; Step S2, data augmentation and processing; Step S3: Construct a segmentation model, which consists of a memory pool, an encoder, a spatiotemporal memory retrieval module, a detail-aware decoder, and a mask correction module; the specific implementation process is as follows: S31 Constructs a memory pool by placing the 1st frame, τ+1st frame, 2τ+1st frame, ..., Tτ+1st frame of the video sequence and their corresponding segmentation labels into the memory pool, where τ is a positive integer and its value ranges from [1, 50] based on the actual length of the video sequence. , The relative position of the current frame to be segmented is indicated by the symbol "". " indicates rounding down; S32 constructs a key encoder, using a convolutional neural network as the key encoder. The memory frame and the current frame sequentially pass through the first four layers of this encoder, with the output of the second layer being... and The outputs of the third layer are respectively and The outputs of the fourth layer are respectively and And process them separately through mapping convolution. and Finally, they obtained and ; S33 constructs a value encoder, which consists of a convolutional neural network and a feature multiplexer. The feature multiplexer consists of one CBAM block and two residual blocks. All memory frames in the memory pool and their segmentation masks are passed through the first four layers of the convolutional neural network to obtain the output of the fourth layer. , With the output of the key encoder After being cascaded, the feature multiplexer processes the data to produce the final Value feature output. ; S34 constructs a spatiotemporal memory retrieval module, which calculates the pairwise similarity between the current video frame to be segmented and the video frames in the memory pool using a non-local attention approach, thereby obtaining the spatiotemporal information of the current frame's Value feature. ; S35 constructs a detail-aware decoder (DAD). The DAD consists of a feature input processing section and a feature output processing section. The feature input processing section comprises one CBAM block and two residual modules. The first residual module consists of three convolutional layers with 512 kernels and a kernel size of 3, and two ReLU activation layers. The second residual module also consists of two convolutional layers with 512 kernels and a kernel size of 3, and two ReLU activation layers. The feature output processing section consists of two upsampling modules and one prediction convolutional module. The first upsampling module… The first module consists of a convolutional layer with 512 kernels and a kernel size of 3, three convolutional layers with 256 kernels and a kernel size of 3, two ReLU activation layers, and a bilinear interpolation module with a scaling factor of 2. The second upsampling module consists of three convolutional layers with 256 kernels and a kernel size of 3, two ReLU activation layers, and a bilinear interpolation module with a scaling factor of 2. The prediction convolutional module consists of a convolutional layer with 1 kernel and a kernel size of 3, a ReLU activation layer, and a bilinear interpolation module with a scaling factor of 2. fq4 obtained in step S32 and fq4 obtained in step S34 Cascade , After the first residual block processing, the number of feature channels is reduced from 1536 to 512. Then, the features are enhanced by the CBAM block. Finally, after further learning by the second residual block, f01 is obtained. f01 and fq3 obtained in step S32 are input into the first upsampling module of the detail-aware decoder to obtain f02; f02 and fq2 obtained in step S32 are input into the second upsampling module of the detail-aware decoder to obtain f03; f03 is input into the prediction convolution module of the detail-aware decoder to obtain the coarse prediction segmentation label. ; S36 constructs a mask correction module, which consists of an uncertain region estimation module, an average pooling layer with a window size of 7, a max pooling layer with a window size of 7, a convolutional layer with 32 kernels and a kernel size of 3, a residual module, and a prediction convolutional layer. The residual module consists of two convolutional layers with 32 kernels and a kernel size of 3 and two ReLU activation layers. The prediction convolutional layer consists of a convolutional layer with 1 kernel and a kernel size of 3 and a ReLU activation layer. S361 To measure the classification uncertainty of each pixel, a coarse probability map output from step S35 is generated. Calculate the corresponding uncertainty region estimation map. : in, Indicates the total number of targets and backgrounds; yes Each pixel in Maximum value in dimension yes Each pixel in The second largest value in the dimension, and In the target dimension The ratio of the highest probability to the second highest probability of a pixel is calculated and normalized to between 0 and 1 using an exponential function; the result will be used as a confidence level to measure whether the pixel segmentation is reliable. The smaller the value, the higher the confidence level of the pixel classification; conversely, the larger the value, the more unreliable the pixel segmentation estimate is, and it will be classified into an uncertain segmentation region. S362 Obtain the rough probability map from step S35. The process, along with step S32, yields fq1, which possesses high-fine-grained information and has a scale similar in size to the coarse probability map. This makes it more suitable for correcting the prediction mask for uncertain region segmentation. For the target... First, average pooling layer and max pooling layer are used for processing. The position of any pixel is obtained by combining fq1 and fq1. Local reference features with target prior : in, In pixels Centered The size of the neighborhood, For the neighborhood Any pixel in For pixels Belongs to the target The probability value, For pixels The local features are obtained by performing the same operation on each pixel of the coarse probability map of each target and background. ; S363 Compare the local feature fq1 obtained in step S32 with the local reference feature obtained in step S362. The similarity between them is determined by concatenating y and fq1, processing them through a mapping convolutional layer, and then using a residual network module to learn and predict local feature similarity, thus obtaining the feature map. : in, It is a cascade at the channel dimension. It is the residual module of the mask correction module. and These are the mapping convolutional layer and the prediction convolutional layer of the mask correction module, respectively; S364 utilizes max pooling layers to combine pixels The maximum probability value within the neighborhood; calculating the probabilistic graph of local optimization. as follows: in, This indicates that the pixel is calculated using the max pooling layer of the mask correction module. The maximum probability value within the neighborhood is obtained by repeating the same operation for each pixel. This value can guide the local optimization mask to optimize towards a target with high confidence. S365. The local optimization probability map obtained in step S364 is used. Add to the rough probability map obtained in step S36 Above; simultaneously, the uncertainty probability diagram obtained through step S361. The local optimization probability map is obtained in step S364. Weighting is applied to focus more on correcting uncertain regions; thus, an accurate video object segmentation prediction is ultimately obtained. as follows: Finally, the segmentation results Bilinear upsampling by a factor of 2 is used to obtain the network output mask image, thus completing the construction of the segmentation model; Step S4: Construct the loss function; Step S5: Train the segmentation model; Step S6, video target segmentation.

2. The semi-supervised video target segmentation method with mask correction as described in claim 1, characterized in that, The specific implementation process of step S1 is as follows: Obtain the video object segmentation dataset and the static image dataset, along with their corresponding segmentation labels. Combine each image in the dataset with its corresponding segmentation label to form an image pair.

3. The semi-supervised video target segmentation method with mask correction as described in claim 2, characterized in that, The specific implementation process of step S2 is as follows: S21. Normalize the static image dataset and corresponding segmentation labels obtained from step S1. To generate synthetic video training samples, repeat the following process: S211 reduces the shorter side of the image pair to w pixels, while proportionally reducing the longer side; then, randomly crop the image pair to... The pixel size, where w represents the width of the cropped image and h represents the height of the cropped image, where both w and h are positive integers; S212 applies random scaling, random horizontal flipping, random color dithering, and random grayscale conversion to the cropped image pair in sequence to obtain the enhanced image pair; S213 Repeat step S212 three times to obtain three augmented samples for the image pair; these three augmented samples form a synthetic video training sample. S22 For each video and its corresponding segmentation label in the video object segmentation dataset obtained from step S1, normalization processing is performed. To generate real video training samples, the following process is repeated: S221 Randomly select three image pairs, which are obtained from the video and have corresponding segmentation labels; S222 reduces the shorter side of the three image pairs to w pixels, while proportionally reducing the longer side; then, it randomly crops the three image pairs to... The pixel size, where the meaning and values ​​of w and h are the same as in step S211; S223 applies random cropping, color jittering, and random grayscale conversion to these three image pairs in sequence to obtain three enhanced image pairs; these three enhanced image pairs form a real video training sample.

4. The semi-supervised video target segmentation method with mask correction as described in claim 1, characterized in that, The specific implementation process of step S4 is as follows: Construct a loss function, where the total loss function is... Using the cross-entropy loss function and the confidence loss function for uncertain regions The sum is defined as follows: in =1, It is 0.5; S41. Construct the cross-entropy loss function The definition is as follows: in It's a real label. It predicts segmentation labels. and These are the height and width of the actual segmentation label, respectively. yes The Middle Line 1 Column pixel values, yes The pixel value in the i-th row and j-th column. =1,2,…, , =1,2,…, log(.) represents the natural logarithm; S42. Construct the confidence loss function for the uncertain region, defined as follows: in This is the probability map of the uncertain region obtained in step S361. This indicates the search for the Euclidean norm.

5. A semi-supervised video target segmentation method with mask correction as described in any one of claims 1, 3, and 4, characterized in that, The specific implementation process of step S5 is as follows: The segmentation model constructed in S3 is trained using the static image-to-video training set obtained in S21. The loss value is obtained according to the loss function constructed in S4, and the parameters within the segmentation model are updated using stochastic gradient descent to obtain a pre-trained model. The pre-trained model is then trained using the real video training set obtained in S22. The loss value is obtained according to the loss function constructed in S4, and the parameters within the model are updated using stochastic gradient descent to obtain a trained segmentation model.

6. The semi-supervised video target segmentation method with mask correction as described in claim 5, characterized in that, The specific implementation process of step S6 is as follows: Obtain the video to be segmented and its segmentation label corresponding to the first frame. Starting from the second frame of the video to be segmented, input them sequentially into the segmentation model trained by S5 and output the segmentation label.

7. A semi-supervised video target segmentation system with mask correction, characterized in that, Performing the steps of any one of claims 1-6 comprises the following units: Video sequence input unit: Given a set of video frames, expand and process them according to the form in step S2 to obtain the input video sequence; The model training unit is used to train a mask-corrected semi-supervised video target segmenter. It consists of a memory pool, an encoder, a spatiotemporal memory reading module, a detail-aware decoder, and a mask correction module. The memory pool module randomly selects three frames as training video frames according to the video sequence frame order at certain intervals. The first frame of the video contains the segmentation mask, the second frame and the following frames are used as query frames to be segmented, and the video sequence before the query frames are all used as memory frames. The feature encoder sequentially extracts the memory key features, query key features, and memory value features of the video frame images. Then, it inputs the memory key features, query key features, and memory value features into the spatiotemporal memory reading module to read the key feature information of the query frame based on the memory frame in order to obtain a more discriminative query value feature. The query frame Value feature is input into the detail-aware decoder to obtain a coarse mask; an uncertain region probability map is obtained based on the coarse mask; then the uncertain region probability map and the coarse mask map are input into the mask correction module to obtain the corrected segmentation prediction mask map of the query frame; finally, the target segmentation is trained by minimizing the cross-entropy loss between the prediction mask map and the real mask map and the uncertainty region confidence loss. In the testing phase, the video target segmentation unit selects one frame from every five frames of the video sequence as described in step S31 and adds it to the memory pool as a memory frame. Starting from the second frame of the video to be segmented, it is sequentially input into the segmentation model trained in step S5 to perform video target segmentation for subsequent frames.

Images