Semi-supervised video target segmentation method and system based on global and local feature fusion

Abstract

The invention discloses a global and local feature fusion-based semi-supervised video target segmentation method and a global and local feature fusion-based semi-supervised video target segmentation system. According to the implementation scheme, the method comprises the following steps: firstly, acquiring required data sets including image and video data sets and corresponding segmentation labels, expanding the image data set, and sampling the video data set to obtain a training data set for training a segmentation model; and secondly, designing a segmentation model, constructing a loss function to train the segmentation model, and finally performing video target segmentation by using the trained segmentation model. According to the invention, a local-global branch fusion module is designed to carry out global context modeling and local detail enhancement on features used for non-local matching, so that the distinguishing capability of the model for distinguishing similar features is improved; therefore, the network can effectively solve the problem of wrong feature matching caused by similar interferents in a semi-supervised video target segmentation task. On this basis, a target scale self-adaptive down-sampling layer is further combined, and information of the same target under different scales is captured through the layer, so that the self-attention calculation complexity is reduced while the feature distinguishability is improved, and the balance between the model segmentation performance and the reasoning prediction efficiency is realized.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method and system for semi-supervised video target segmentation by fusion of global and local features. Background Art

[0002] Video Object Segmentation (VOS) has always been a hot research issue in the field of computer vision and video editing. Its main task is to perform high-quality segmentation of target instances in the entire input video sequence and obtain pixel-level masks of target objects, so as to finely segment the target from the background image, so that the model can locate the video target more accurately. At the same time, VOS not only provides the location information of the target, but also gives detailed semantic information of the target, which is crucial for understanding the video content, identifying different target categories, and inferring the behavior and intention of the target. For example, in scenarios related to autonomous driving, vehicles can rely on segmentation models to identify and locate various elements, such as other vehicles, pedestrians, traffic signs, and road obstacles. This segmentation technology enables vehicles to make intelligent decisions, such as when to stop, slow down, change lanes, or avoid obstacles. In video surveillance systems, segmentation models can be used to detect and track specific targets, such as pedestrians, vehicles, or other suspicious behaviors, and can trigger alarms or take actions based on specific scenarios. This helps improve the accuracy and efficiency of monitoring systems, as well as the real-time response to events.

[0003] In recent years, with the rapid development of deep learning and convolutional networks, deep convolutional neural networks have been widely used in VOS tasks. Researchers prefer video object segmentation methods based on spatiotemporal memory networks, which not only have an advantage in segmentation speed, but also achieve good accuracy. In 2019, Seoung Wug Oh et al. first proposed a spatiotemporal memory network (Seoung Wug Oh, Joon-Young Lee, Ning Xu, Seon Joo Kim. Video object segmentation using space-time memory networks. ICCV 2019: 9225-9234.), which uses a non-local attention mechanism to calculate the feature matching of the current frame and the historical frame, and introduces an external memory pool to continuously store the feature representation of the historical frames and their segmentation results, thereby achieving long-term matching. Due to the excellent performance and high efficiency of memory networks, such methods currently dominate the field of semi-supervised VOS.

[0004] However, methods based on spatiotemporal memory networks have the problem of mismatching of similar targets. In recent years, some methods based on spatiotemporal memory networks have attempted to solve the problem of mismatching of similar targets in order to achieve accurate video target segmentation in complex scenes. For example, Hongje Seong et al. proposed the KMN method (Hongje Seong, Junhyuk Hyun, Euntai Kim. Kernelized memory network for video object segmentation. ECCV 2020: 629-645) and Haozhe Xie et al. proposed the RMNet method (Haozhe Xie, Hongxun Yao, Shangchen Zhou, et al. Efficient regional memory network for video object segmentation. CVPR 2021: 1286-1295), both of which use Gaussian kernels and hard clipping strategies for local-to-local matching instead of global-to-global matching in STM. Li Hu et al. proposed the LCM method (Hu L, Zhang P, Zhang B, Pan P, Xu Y, Jin R. Learning position and target consistency for memory-based video objects segmentation. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2021, 4144–4154), which uses position encoding to perform spatial constraints and enhances the spatial position information of the target to reduce the probability of false matching of similar targets in the non-local matching mechanism. Yong Liu et al. enhanced the low-frequency components of the features in the encoding stage to improve the robustness of non-local matching and proposed the GSFM method (Liu Y, Yu R, Wang J, Zhao X, Wang Y, Tang Y, Yang Y. Global spectral filter memory network for video object segmentation. In European Conference on Computer Vision, Springer 2022, 648–665). Although these memory matching-based methods have achieved good performance, they ignore the importance of global context information modeling and local feature enhancement in extracting strong representation learning features, and are still prone to mutual interference between similar local features. Summary of the invention

[0005] In view of the shortcomings of the above-mentioned prior art, the present invention provides a semi-supervised video target segmentation method and system with global and local feature fusion, which takes a video sequence and the segmentation label of its first frame as input, extracts the feature information of the video frame through a Key encoder based on a convolutional neural network, and extracts the feature information of the video frame and its segmentation label through a Value encoder; designs and implements a local-global branch fusion module (Lobal-Global-Branch Fusion Module, LGBFM), which is used to model the global dependency of the Key encoder features while compensating for local details, and improves the robustness of non-local matching in VOS by learning a stronger global feature representation; the spatiotemporal memory reading module is used to obtain the Value feature of the current query frame; the decoder decodes the Value feature of the current query frame to finally obtain a refined segmentation label.

[0006] In order to achieve the above object, the present invention provides a method and system for semi-supervised video object segmentation by global and local feature fusion, comprising the following steps:

[0007] S1. Obtain the required data sets, including image and video data sets;

[0008] S2, expanding the image data set and sampling the video data set to obtain a training data set for training the segmentation model;

[0009] S3. Design a segmentation model, which consists of an encoder, a local-global-branch fusion module (LGBFM), a spatiotemporal memory reading module, a decoder, and a spatiotemporal memory pool;

[0010] S4. Design loss function for training segmentation model;

[0011] S5, use the training data set obtained in step S2 and the loss function in S4 to train the segmentation model in S3;

[0012] S6. Use the segmentation model trained in step S5 to perform semi-supervised video target segmentation.

[0013] The present invention provides an end-to-end trainable neural network architecture and system for semi-supervised video object segmentation, including a video sequence input module: given a video sequence At the same time, given the segmentation label Y1 of the first frame in the video, the model automatically segments all remaining video frames according to the foreground target marked in Y1. The model training module trains the segmentation network by minimizing the joint loss, including the cross entropy loss and the dice loss. In the video target segmentation module, in the model inference stage, as described in step S3, one frame is selected for every 5 frames of the video sequence segmented and added to the memory pool as a memory frame, and starting from the second frame of the video to be segmented, it is sequentially input into the segmentation model trained in step S5 to automatically segment the video targets of subsequent frames.

[0014] Compared with the prior art, the present invention has the following beneficial effects:

[0015] The present invention considers the above problems and adopts a local-global branch fusion module (Lobal-Global-Branch Fusion Module, LGBFM) in the video target segmentation network to perform global context modeling and local detail enhancement on the features used for non-local matching, thereby improving the model's ability to distinguish similar features, so that the network can effectively solve the problem of erroneous feature matching caused by similar interferers in the semi-supervised video target segmentation task. And on this basis, it further combines the object scale adaptive downsampling (Object Scale Adaptive Downsampling, OSAD), through which OSAD captures the information of the same target at different scales, while improving the feature discriminability, it also reduces the computational complexity of self-attention, thereby improving the model segmentation performance while reducing the consumption of inference prediction time. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 Flowchart of the semi-supervised video object segmentation method in the present invention patent;

[0018] Figure 2 The overall structure diagram of the video object segmentation network in the patent of this invention;

[0019] Figure 3 The global-local branch structure diagram of the network in the present invention patent;

[0020] Figure 4 The feature matching result diagram of the network for similar targets in the patent of this invention;

[0021] Figure 5 Schematic diagram of the segmentation results of similar targets by the network in this patent. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in each embodiment of the present invention described below can be combined with each other as long as they do not conflict with each other. The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0023] Figure 1 The flowchart of the semi-supervised video target segmentation method and system with global feature enhancement in the patent of this invention is shown as follows:

[0024] Step S1, obtaining static image and video target segmentation datasets and their segmentation labels, the specific steps are as follows:

[0025] S101, obtaining static image data set I s = {I 1 ,I 2 ,…,I N} and Video Object Segmentation Dataset I v = {I 1 ,I 2 ,...,I M} and its corresponding segmentation label G s = {G 1 ,G 2 ,…,G N} and G v = {G 1 ,G 2 ,…,G M}, each image in the dataset and its corresponding segmentation label form an image-segmentation label pair set P s ={(I 1 ,G 1 ),(I 2 ,G 2 ),…,(I N ,G N )} and P v ={(I 1 ,G 1 ),(I 2 ,G 2 ),…,(I M ,G M )}.

[0026] Step S2, expanding the image data set and sampling the video data set to obtain a training data set for training the segmentation model, specifically includes the following steps:

[0027] S201, the set P obtained in step S1 s For each image-segmentation label pair in The enhanced label pair is obtained after normalization, cropping, and affine transformation. Among them, affine transformation includes random scaling, random horizontal flipping, random color jittering and random grayscale change. A total of 3 affine transformations are performed to obtain a pseudo video dataset through static images.

[0028] S202: the set P obtained in step S1 v Each image-segmentation label pair in is normalized, cropped, and affine transformed to obtain an enhanced label pair Affine transformation includes random scaling, random horizontal flipping, random color jittering, and random grayscale changes. Three frames are sampled at equal intervals from the same video sequence to obtain a video dataset.

[0029] Step S3, designing a segmentation model, which consists of an encoder, a local-global-branch fusion module (LGBFM), a spatiotemporal memory reading module, a decoder, and a spatiotemporal memory pool; Figure 2 The following is a diagram showing the structure of the segmentation model system in the present invention. The specific steps are as follows:

[0030] S301, construct a Key encoder, use a convolutional neural network as the Key encoder, the encoder takes the current query video frame I to be segmented q As input, it outputs a feature map with a stride of 16 Where 1024, H, and W represent the height and width of the original image and the number of feature channels of the feature map, respectively. In the following text, h = H / 16 and w = W / 16. Please note that in the VOS task, the model will share the Key encoder output features. In other words, if the network wants to remember the query frame, it will share the Key encoder output features. Q Stored in the space-time memory pool as K M .

[0031] S302, construct a Value encoder, and use a convolutional neural network as the Value encoder. The encoder is based on the memory video frame I m and its segmentation result Y m As input, it outputs a feature map with a stride of 16 At the same time, the I obtained in S301 is reused m The Key features of V are used to enrich the output features of the Value encoder. M With K M Cascaded and processed by two residual blocks, the final Value feature is obtained, denoted as

[0032] S303, constructing LGBFM module, Figure 3 The figure shows the schematic diagram of the LGBFM module structure in the patent of this invention. The module includes a global branch, a local branch and a fully connected feed-forward network (FFN). The global branch consists of an object scale adaptive downsampling (OSAD) layer, a multi-head self-attention (MHSA) layer, and a 1×1 convolution layer. The local branch consists of a depth-separable convolution layer with a convolution kernel size of 3×3 and a 1×1 standard convolution layer. A batch normalization layer is used after each convolution layer to process the features of the convolution layer output. The LGBFM module uses the K obtained by S301 Q As input, we model K through the global branch Q The global context information of K is used to enhance its context-aware features. The local branch is used to enhance K Q Spatial details, global and local feature fusion within the module, and finally output enhanced Key features This module performs global context modeling and local spatial detail enhancement on the features extracted by the Key encoder, so that the network can learn stronger Key feature representations, thereby improving the model's ability to distinguish similar targets, and ultimately achieving more accurate video target segmentation in scenarios with similar distractors. The specific implementation details are as follows:

[0033] S3031, obtained from step S301 Perform a 3×3 convolution operation to reduce the number of feature channels and use it as the input feature of the global branch. This branch first obtains the features used for self-attention calculation and Then, the target scale adaptive downsampling (OSAD) layer is used to calculate the optimal downsampling coefficients of K and V on different self-attention heads, and K and V are downsampled according to the coefficients. The specific calculation process is as follows:

[0034] Q i =QW i Q ,

[0035] α i =softmax(W i 2 δ(W i 1 AP(Q i ))),

[0036] K i =SR(K,R i )max(α i )W i K ,

[0037] V i =SR(V,R i )max(α i )W i V ,

[0038] Where W is the linear transformation matrix, α i is the output of the OSAD layer of the ith attention head, max(α i ) represents the optimal downsampling coefficient of K and V on the i-th attention head. By capturing the information of the same target at different scales through OSAD, the time complexity of subsequent multi-head self-attention calculation is reduced while improving feature discriminability.

[0039] S3032, obtained from step S3031 Perform multi-head self-attention calculations to model the global context of features and finally obtain the global enhanced feature F global , the specific calculation process is as follows:

[0040] F global =MultiHead(Q,K,V)=Concat(H1,...,H d )W O ,

[0041] H i =Attention(Q i ,K i ,V i ),

[0042]

[0043] Among them, H i is the i-th attention head, and d is the total number of attention heads.

[0044] S3033, K obtained from step S301 Q Perform a 3×3 convolution operation to reduce the number of feature channels and use it as the input feature of the local branch. The local branch uses a depthwise separable convolutional layer with a kernel size of 3×3 and a standard convolutional layer of 1×1 to enhance the input feature F. in The spatial details of each convolution layer are processed by batch normalization. Finally, the local enhanced feature F is obtained. local The specific calculation is as follows:

[0045] F local =BN(Conv(ReLU(F))),

[0046] F = BN(DWConv(F in )),

[0047] Among them, DWConv is a depthwise separable convolution, BN represents a batch normalization layer, Conv is a 1×1 convolution, and ReLU is a nonlinear activation function. The local branch uses the depthwise separable aggregation of local details and the standard convolution to enhance the representation of the feature map, which can effectively reduce the local noise introduced by the self-attention calculation in the global branch and achieve more robust feature modeling.

[0048] S3034, the global feature F obtained from steps S3032 and S3031 global and local features F local First, the Hadamard product is used to achieve the feature fusion of the global and local branches, and the input feature F is accumulated in Perform feature enhancement to obtain fusion features The specific calculation process is as follows:

[0049] F fusion =F global ⊙F local +F in ,

[0050] Among them, ⊙ represents the HadamardProduct calculation.

[0051] S304, constructing a spatiotemporal memory reading module, given a query frame I q Enhanced Key Features T pairs of memory frames and their segmentation results in the memory pool Enhanced Key and Value features This module first calculates the and The normalized similarity matrix A∈[0,1] Thw×hw , Figure 4 is the matching result diagram of the segmentation model invented in this patent. Then, Perform matrix multiplication and weighted summation to finally obtain the reconstructed query frame Value feature The specific calculation process is as follows:

[0052]

[0053] in, represent The eigenvector at the i-th position in , represent The feature vector of the jth position in the memory frame. D(·,·) is the L2 distance metric function. The normalized weight A(i,j)∈[0,1] represents the similarity between the i-th pixel in the memory frame and the j-th pixel in the query frame.

[0054] S305. Construct a decoder. The decoder consists of two upsampling modules, wherein the first upsampling module consists of four convolutional layers, two ReLU activation layers and a bilinear interpolation, and the second upsampling module consists of three convolutional layers, two ReLU activation layers and a bilinear interpolation. The query frame Value feature V obtained in S304 Q As the input of the decoder, it is gradually upsampled at a ratio of two in the decoder. At the same time, a skip connection is used to combine the key features of the query frame obtained in S301 The decoder outputs a query frame segmentation label with a stride of 4, which is finally restored to the original resolution through bilinear upsampling to obtain the segmentation result of the query frame.

[0055] S306: Construct a spatiotemporal memory pool. First, store the first frame of the video sequence to be segmented and its segmentation mask {(I1, Y1)} in the memory pool. In the subsequent segmentation process, the video frame and its corresponding segmentation label {(I1, Y1)} are stored in the memory pool every r frames. 1+r ,Y 1+r ),(I 1+2r ,Y 1+2r ),…,(I 1+nr ,Y 1+nr )} is stored in the memory pool to save the temporal context information of the video sequence and guide the automatic segmentation of subsequent video frames. L is the total length of the video sequence, symbol Indicates rounding down.

[0056] Step S4, design a loss function for training the segmentation model. The loss function L total It is composed of cross entropy loss and Dice loss, which are defined as follows:

[0057] L total =λ1L BCE +λ2L Dice ,

[0058] Where λ1=λ2=1, and the Dice loss calculation is defined as follows:

[0059]

[0060] Among them, p i represents the predicted value, q i represents the true value, i represents the i-th pixel. κ is a smoothing term used to avoid the zero division problem.

[0061] Step S5: training the segmentation model. Using the static image pseudo video training set obtained in S201 Train the segmentation model constructed in S3, obtain the loss value according to the loss function constructed in S4, and use the stochastic gradient descent method to update the parameters in the segmentation model until convergence to obtain a pre-trained model; use the real video training set obtained in S202 The pre-trained model is trained to obtain the loss value according to the loss function constructed by S4, and the parameters in the model are updated using the stochastic gradient descent method until convergence, and finally a trained segmentation model is obtained.

[0062] Step S6, obtain the segmentation label corresponding to the video sequence to be segmented and its first frame, and input them into the segmentation model trained in S5 in sequence starting from the second frame of the video to be segmented. The model will automatically segment and obtain the segmentation label of each input frame. Figure 5 Schematic diagram of video object segmentation labels of the segmentation model invented in this patent.

[0063] According to another aspect of the present application, a global feature enhanced semi-supervised video object segmentation system is also provided, comprising the following modules:

[0064] Video sequence input module: given a video sequence At the same time, given the segmentation label Y1 of the first frame in the video, the model automatically segments all remaining video frames according to the foreground targets marked in Y1.

[0065] Model training module: used to train a semi-supervised video object segmenter with global and local feature fusion, including feature encoder, local-global-branch fusion module (LGBFM), spatiotemporal memory reading module, decoder and spatiotemporal memory pool module. Training video dataset Each video sequence consists of 3 frames, where the first frame of the video has a segmentation label, the second frame and the following frames are used as query frames to be segmented, and the video sequences before the query frames are used as memory frames. For each query frame, the Key encoder first extracts the Key feature K from it Q , and use LGBFM to enhance the global-local features and output the enhanced Key features At the same time, the model will share the output features of the Key encoder and LGBFM. In other words, if the network wants to remember the query frame, it will use K Q and Stored in the memory pool as K M and The Value encoder extracts the Value feature V of the memory frame M , and then reuse K M Get the enhanced Value feature Then will and As the input of the spatiotemporal memory reading module, the Value feature V of the query frame is obtained Q . V Q The segmentation prediction label map of the query frame can be obtained by inputting it into the decoder. Finally, the video object segmentation model is trained by minimizing the joint loss of the predicted label map and the true label map.

[0066] Video target segmentation module: In the model inference stage, as described in step S306, one frame is selected from every 5 frames of the video sequence to be segmented and added to the memory pool as a memory frame. Starting from the second frame of the video to be segmented, it is input into the segmentation model trained in step S5 in sequence to automatically segment the video targets of subsequent frames.

[0067] The system is used to implement the functions of the methods in the above-mentioned embodiments. The specific implementation steps of the methods involved in the system modules have been described in the methods and will not be repeated here.

[0068] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the inventive concept of the present invention, or directly / indirectly applied in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. A semi-supervised video target segmentation method and system based on global and local feature fusion, characterized in that: The method is executed by a computer and comprises the following steps: S1. Obtain the required data sets, including image and video data sets; S2, expanding the image data set and sampling the video data set to obtain a training data set for training the segmentation model; S3. Design a segmentation model, which consists of an encoder, a local-global-branch fusion module (LGBFM), a spatiotemporal memory reading module, a decoder, and a spatiotemporal memory pool; S4. Design loss function for training segmentation model; S5, use the training data set obtained in step S2 and the loss function in S4 to train the segmentation model in S3; S6. Use the segmentation model trained in step S5 to perform semi-supervised video target segmentation.

2. The step S1 of acquiring a data set as claimed in claim 1, characterized in that: The specific implementation process of S1 is as follows: S101, obtaining static image data set I s = {I 1 ,I 2 ,...,I N } and Video Object Segmentation Dataset I v = {I 1 ,I 2 ,...,I M } and its corresponding segmentation label G s = {G 1 ,G 2 ,...,G N } and G v = {G 1 ,G 2 ,…,G M }, each image in the dataset and its corresponding segmentation label form an image-segmentation label pair set P s ={(I 1 ,G 1 ),(I 2 ,G 2 ),…,(I N ,G N )} and P v ={(I 1 ,G 1 ),(I 2 ,G 2 ),…,(I M ,G M )}.

3. The step S2 of data expansion and processing as claimed in claim 1, characterized in that: The specific implementation process of S2 is as follows: S201, the set P obtained in step S1 s For each image-segmentation label pair in The enhanced label pair is obtained after normalization, cropping, and affine transformation. Among them, affine transformation includes random scaling, random horizontal flipping, random color jittering and random grayscale change. A total of 3 affine transformations are performed to obtain a pseudo video dataset through static images. S202: the set P obtained in step S1 v Each image-segmentation label pair in is normalized, cropped, and affine transformed to obtain an enhanced label pair Affine transformation includes random scaling, random horizontal flipping, random color jittering, and random grayscale changes. Three frames are sampled at equal intervals from the same video sequence to obtain a video dataset.

4. The step S3 of constructing a segmentation model as claimed in claim 1, characterized in that: The specific implementation process of S3 is as follows: S301, construct a Key encoder, use a convolutional neural network as the Key encoder, the encoder takes the current query video frame I to be segmented q As input, it outputs a feature map with a step size of 16, which is used as the Key feature K of the current query video frame to be segmented Q . S302, construct a Value encoder, and use a convolutional neural network as the Value encoder. The encoder is based on the memory video frame I m and its segmentation result Y m As input, it outputs a feature map V with a stride of 16 M , while multiplexing I m The Value feature is finally obtained from the Key feature, denoted as S303, constructing an LGBFM module, which includes a global branch, a local branch and a fully connected feedforward network (FFN). The global branch consists of an Object Scale Adaptive Downsampling (OSAD) layer, a Multi-Head Self-Attention (MHSA) layer, and a 1×1 convolution layer. The local branch consists of a depth-wise separable convolution layer with a convolution kernel size of 3×3 and a 1×1 standard convolution layer. A batch normalization layer is used after each convolution layer to process the features of the convolution layer output. The LGBFM module uses the K obtained by S301 Q As input, we model K through the global branch Q The global context information of K is used to enhance its context-aware features. The local branch is used to enhance K Q Spatial details, global and local feature fusion within the module, and finally output enhanced Key features This module performs global context modeling and local spatial detail enhancement on the features extracted by the Key encoder. The network can learn stronger Key feature representations, thereby improving the model's ability to distinguish similar targets, and ultimately achieving more accurate video target segmentation in scenarios with similar distractors. S304, constructing a spatiotemporal memory reading module, given a query frame I q T pairs of memory frames and their segmentation results in the memory pool This module first calculates the normalized similarity matrix A between the query frame and the memory frame in a non-local attention manner, and then Perform matrix multiplication and weighted summation to finally obtain the reconstructed query frame Value feature V Q . S305. Construct a decoder. The decoder consists of two upsampling modules, wherein the first upsampling module consists of four convolutional layers, two ReLU activation layers and a bilinear interpolation, and the second upsampling module consists of three convolutional layers, two ReLU activation layers and a bilinear interpolation. The query frame Value feature V obtained in S304 Q As the input of the decoder, it is gradually upsampled at a ratio of two in the decoder. At the same time, a skip connection is used to combine the Key feature K of the query frame obtained in S301 Q The decoder outputs a query frame segmentation label with a stride of 4, which is finally restored to the original resolution through bilinear upsampling to obtain the segmentation result of the query frame. S306: Construct a spatiotemporal memory pool. First, store the first frame of the video sequence to be segmented and its segmentation mask {(I1, Y1)} in the memory pool. In the subsequent segmentation process, the video frame and its corresponding segmentation label {(I1, Y1)} are stored in the memory pool every r frames. 1+r ,Y 1+r ),(I 1+2r ,Y 1+2r ),…,(I 1+nr ,Y 1+nr )} is stored in the memory pool to save the temporal context information of the video sequence and guide the automatic segmentation of subsequent video frames. L is the total length of the video sequence, symbol Indicates rounding down.

5. In step S303 as described in claim 4, the LGBFM module mainly comprises a global branch and a local branch to perform a Key feature K of the query frame. Q Perform global and local feature fusion head, characterized by: The specific implementation details of S303 are as follows: S3031, obtained from step S301 Perform a 3×3 convolution operation to reduce the number of feature channels and use it as the input feature of the global branch. This branch first obtains the features used for self-attention calculation and Then, the target scale adaptive downsampling (OSAD) layer is used to calculate the optimal downsampling coefficients of K and V on different self-attention heads, and K and V are downsampled according to the coefficients. The specific calculation process is as follows: Where W is the linear transformation matrix, α i is the output of the OSAD layer of the ith attention head, max(α i ) represents the optimal downsampling coefficient of K and V on the ith attention head. OSAD captures the information of the same target at different scales, which improves feature discriminability and reduces the time complexity of subsequent multi-head self-attention calculations. S3032, obtained from step S3031 Perform multi-head self-attention calculations to model the global context of features and finally obtain the global enhanced feature F global , the specific calculation process is as follows: F global =MultiHead(Q,K,V)=Concat(H1,...,H d )W O , H i =Attention(Q i ,K i ,V i ), Among them, H i is the i-th attention head, and d is the total number of attention heads. S3033, K obtained from step S301 Q Perform a 3×3 convolution operation to reduce the number of feature channels and use it as the input feature of the local branch. The local branch uses a depthwise separable convolutional layer with a kernel size of 3×3 and a standard convolutional layer of 1×1 to enhance the input feature F. in The spatial details of each convolution layer are processed by batch normalization. Finally, the local enhanced feature F is obtained. local The specific calculation is as follows: F local =BN(Conv(ReLU(F))), F=BN(DWConv(F in )), Among them, DWConv is a depthwise separable convolution, BN represents a batch normalization layer, Conv is a 1×1 convolution, and ReLU is a nonlinear activation function. The local branch uses the depthwise separable aggregation of local details and the standard convolution to enhance the representation of the feature map, which can effectively reduce the local noise introduced by the self-attention calculation in the global branch and achieve more robust feature modeling. S3034, the global feature F obtained from steps S3032 and S3031 global and local features F local First, the Hadamard product is used to achieve the feature fusion of the global and local branches, and the input feature F is accumulated in Perform feature enhancement to obtain fusion features The specific calculation process is as follows: F fusion =F global ⊙F local +F in , Among them, ⊙ represents the HadamardProduct calculation.

6. The step S4 of claim 1 constructs a loss function, wherein: The specific implementation process of S4 is as follows: Construct the loss function, where the loss function L total It is composed of cross entropy loss and Dice loss, which are defined as follows: L total =λ1L BCE +λ2L Dice , Where λ1=λ2=1, and the Dice loss calculation is defined as follows: Among them, p i represents the predicted value, q i represents the true value, i represents the i-th pixel. κ is a smoothing term used to avoid the zero division problem.

7. The step S5 of training the segmentation model as claimed in claim 1, characterized in that: The specific implementation process of S5 is as follows: Static image pseudo video training set obtained using S201 Train the segmentation model constructed in S3, obtain the loss value according to the loss function constructed in S4, and use the stochastic gradient descent method to update the parameters in the segmentation model until convergence to obtain a pre-trained model; use the real video training set obtained in S202 The pre-trained model is trained, the loss value is obtained according to the loss function constructed by S4, and the parameters in the model are updated using the stochastic gradient descent method until convergence, and finally a trained segmentation model is obtained.

8. The step S6 of video object segmentation as claimed in claim 1, characterized in that: The specific implementation process of S6 is as follows: Get the segmentation label corresponding to the video sequence to be segmented and its first frame, and input them into the segmentation model trained by S5 starting from the second frame of the video to be segmented. The model will automatically segment and obtain the segmentation label of each input frame.

9. A semi-supervised video target segmentation method and system based on global and local feature fusion, characterized in that: Contains the following modules: Video sequence input module: given a video sequence At the same time, given the segmentation label Y1 of the first frame in the video, the model automatically segments all remaining video frames according to the foreground targets marked in Y1. Model training module: used to train a semi-supervised video object segmenter with global and local feature fusion, including feature encoder, local-global-branch fusion module (LGBFM), spatiotemporal memory reading module, decoder and spatiotemporal memory pool module. Training video dataset Each video sequence consists of 3 frames, where the first frame of the video has a segmentation label, the second frame and the following frames are used as query frames to be segmented, and the video sequences before the query frames are used as memory frames. For each query frame, the Key encoder first extracts the Key feature K from it Q , and use LGBFM to enhance the global-local features and output the enhanced Key features At the same time, the model will share the output features of the Key encoder and LGBFM. In other words, if the network wants to remember the query frame, it will share the K Q and Stored in the memory pool as K M and The Value encoder extracts the Value feature V of the memory frame M , and then reuse K M Get the enhanced Value feature Then will and As the input of the spatiotemporal memory reading module, the Value feature V of the query frame is obtained Q . V Q The segmentation prediction label map of the query frame can be obtained by inputting it into the decoder. Finally, the video object segmentation model is trained by minimizing the joint loss of the predicted label map and the true label map. Video target segmentation module: In the model inference stage, as described in step S306, one frame is selected from every 5 frames of the video sequence to be segmented and added to the memory pool as a memory frame. Starting from the second frame of the video to be segmented, it is input into the segmentation model trained in step S5 in sequence to automatically segment the video targets of subsequent frames.

Images