Foreground object segmentation method, apparatus, device, and storage medium based on contrastive learning

By generating predictive masks and global feature vectors for video frames through object segmentation and discrimination networks, and dynamically updating memory features, the problem of memory networks being unable to adjust in a timely manner in existing technologies is solved, achieving higher accuracy and robustness in video foreground object segmentation.

CN119006826BActive Publication Date: 2026-11-17PENG CHENG LAB
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411152736.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-21
Publication Date
2026-11-17
Estimated Expiration
2044-08-21

AI Technical Summary

Technical Problem

Existing video foreground object sequence segmentation methods based on memory networks and feature matching struggle to adjust the memory in a timely manner according to the changing characteristics of objects in the video. This leads to the loss of key features, making it impossible to accurately judge the changes in objects, effectively fuse redundant features, or enhance newly emerging key features, thus making it difficult to achieve effective segmentation of target objects in complex video scenes.

Method used

The target segmentation network generates a predicted mask for the current video frame, the target discrimination network calculates the global feature vector, and determines the feature update method based on vector similarity and similarity threshold. The memory features in the memory network are dynamically adjusted to achieve short-term or long-term memory updates and capture the key features of the foreground object in the process of change.

Benefits of technology

It improves the accuracy and robustness of video foreground object sequence segmentation, enabling more accurate segmentation of target objects in various scenarios, and dynamically adjusts the memory network to adapt to object changes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006826B_ABST
    Figure CN119006826B_ABST
Patent Text Reader

Abstract

This application discloses a foreground object segmentation method, apparatus, device, and storage medium based on contrastive learning, relating to the field of video processing technology. The method includes: generating a current prediction mask for the current video frame through a target segmentation network; obtaining a global feature vector for the current video frame through a target discriminant network; determining a feature update method based on the global feature vector and an initial feature vector, combined with a similarity threshold; updating the memory features according to the feature update method and the current prediction mask; and segmenting the target object in subsequent video frames in the target video according to the updated memory features until object segmentation in each video frame is complete. Through this method, the changing state of foreground objects is discerned within a unified segmentation framework, key features of foreground objects during the changing process are captured, and the memory features stored in the memory network are dynamically adjusted, thereby improving the accuracy and robustness of foreground object sequence segmentation in various scenarios.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a method, apparatus, device and storage medium for foreground object segmentation based on contrastive learning. Background Technology

[0002] Segmentation methods based on memory networks and feature matching have become a research hotspot in video foreground object sequence segmentation in recent years. Based on multi-level memory mechanisms and complex matching patterns, some researchers have proposed memory matching models, which have received widespread attention in the field of video foreground object sequence segmentation. However, current segmentation methods based on memory networks and feature matching all use fixed intervals to update the memory network. This makes it difficult to adjust the memory in a timely manner according to the changing characteristics of the object in the video, easily losing key features that appear during video changes, which is crucial for effectively segmenting targets that have undergone significant changes. Furthermore, because it cannot accurately judge the changes in the object, the memory network cannot promptly fuse redundant features when the foreground object changes little, nor can it specifically enhance newly emerging key features when the foreground object changes significantly, making it difficult to achieve effective segmentation of target objects in complex video scenes. Summary of the Invention

[0003] The main objective of this application is to provide a method, apparatus, device, and storage medium for foreground object segmentation based on contrastive learning, aiming to solve the technical problem of how to improve the accuracy and robustness of video foreground object sequence segmentation.

[0004] To achieve the above objectives, this application proposes a foreground object segmentation method based on contrastive learning, wherein the foreground object segmentation method based on contrastive learning includes:

[0005] The target segmentation network generates the current prediction mask for the current video frame based on the current video frame features corresponding to the current video frame and the memory features stored in the memory network.

[0006] The target discrimination network obtains the global feature vector of the current video frame based on the currently extracted features corresponding to the current video frame;

[0007] Similarity is calculated based on the global feature vector and the initial feature vector, and the feature update method is determined based on the vector similarity and the similarity threshold.

[0008] The memory features are updated according to the feature update method and the current prediction mask to obtain the updated memory features;

[0009] Based on the updated memory features, target object segmentation is performed on each video frame of the target video until object segmentation of each video frame is completed.

[0010] In one embodiment, the step of generating the current prediction mask of the current video frame through the target segmentation network based on the current video frame features corresponding to the current video frame and the memory features stored in the memory network includes:

[0011] The current video frame features are obtained by extracting features from the feature extraction network in the target segmentation network.

[0012] The target enhancement features are obtained by performing a matching operation based on the current video frame features and the memory features stored in the memory network;

[0013] The target enhancement features are input into the decoder in the target segmentation network to generate the current prediction mask for the current video frame.

[0014] In one embodiment, the step of performing a matching operation based on the current video frame features and the memory features stored in the memory network to obtain the target enhancement features includes:

[0015] The number of feature channels of the current video frame features is compressed to generate a target query;

[0016] The feature keys and target values ​​are determined based on the memory features stored in the memory network;

[0017] A similarity matrix is ​​obtained by performing a similarity calculation based on the target query and the feature key;

[0018] Feature generation is performed based on the similarity matrix and the target value to obtain target enhancement features.

[0019] In one embodiment, the step of obtaining the global feature vector of the current video frame through the target discrimination network based on the currently extracted features corresponding to the current video frame includes:

[0020] The feature extraction network in the target discrimination network is used to extract features from the current video frame to obtain the current extracted features corresponding to the current video frame.

[0021] The extracted features are mapped to obtain a first query, a first key, and a first value;

[0022] The target activation feature is obtained based on the first query, the first key, and the first value;

[0023] Semantic activation is performed based on the second query, second key, and second value corresponding to the target activation feature to obtain the semantic activation feature;

[0024] The global feature vector of the current video frame is obtained based on the semantic activation features.

[0025] In one embodiment, before the step of calculating similarity based on the global feature vector and the initial feature vector, and determining the feature update method based on the vector similarity and a similarity threshold, the method further includes:

[0026] Multiple extracted frames are obtained by randomly selecting video frames from multiple video segments corresponding to the target video.

[0027] The global feature vector of each extracted frame is determined by the target discrimination network.

[0028] The similarity between adjacent video frames is calculated based on the global feature vector of each extracted frame to obtain multiple target similarities.

[0029] The similarity threshold is obtained by calculating the mean of the similarity of each target.

[0030] In one embodiment, the step of updating the memory features according to the feature update method and the current prediction mask to obtain the updated memory features includes:

[0031] When the feature update method is short-term memory update, the initial encoded features of the segment are obtained;

[0032] The current long-term memory is determined based on the memory features, and the current coding features are obtained based on the current prediction mask and the current video frame features;

[0033] Based on the current encoded features and the initial encoded features of the segment, feature concatenation is performed to obtain concatenated features;

[0034] Multi-scale fusion of the spliced ​​features yields the target short-term memory;

[0035] The updated memory features are obtained based on the target short-term memory and the current long-term memory.

[0036] In one embodiment, the step of updating the memory features according to the feature update method and the current prediction mask to obtain the updated memory features includes:

[0037] When the feature update method is long-term memory update method, the current short-term memory is determined according to the memory features, and the current coding features are obtained according to the current prediction mask and the current video frame features;

[0038] A similarity matrix is ​​obtained by calculating the similarity based on the current encoding features and the initial encoding features of the segment;

[0039] Feature processing is performed based on the similarity matrix and the initial encoded features of the segment to determine the adjustment weights;

[0040] Based on the adjusted weights, feature processing is performed on the current encoded features to obtain the target long-term memory;

[0041] The updated memory features are obtained based on the target long-term memory and the current short-term memory.

[0042] Furthermore, to achieve the above objectives, this application also proposes a foreground object segmentation device based on contrastive learning, the foreground object segmentation device based on contrastive learning comprising:

[0043] The generation module is used to generate the current prediction mask of the current video frame through the target segmentation network based on the current video frame features corresponding to the current video frame and the memory features stored in the memory network.

[0044] The processing module is used to obtain the global feature vector of the current video frame based on the currently extracted features corresponding to the current video frame through the target discrimination network;

[0045] The calculation module is used to calculate the similarity based on the global feature vector and the initial feature vector, and to determine the feature update method based on the vector similarity and the similarity threshold.

[0046] The update module is used to update the memory features according to the feature update method and the current prediction mask to obtain the updated memory features;

[0047] The segmentation module is used to segment the target objects in each video frame of the target video according to the updated memory features until the object segmentation of each video frame is completed.

[0048] Furthermore, to achieve the above objectives, this application also proposes a foreground object segmentation device based on contrastive learning, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the foreground object segmentation method based on contrastive learning as described above.

[0049] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the foreground object segmentation method based on contrastive learning as described above.

[0050] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the foreground object segmentation method based on contrastive learning as described above.

[0051] This application generates a current prediction mask for the current video frame using a target segmentation network based on the current video frame features and memory features stored in a memory network. A target discrimination network obtains a global feature vector for the current video frame based on the currently extracted features. Similarity is calculated between the global feature vector and an initial feature vector, and a feature update method is determined based on vector similarity and a similarity threshold. The memory features are updated according to the feature update method and the current prediction mask to obtain updated memory features. Target object segmentation is performed on each video frame of the target video using the updated memory features until object segmentation of each video frame is complete. Through this method, the changing state of foreground objects is determined within a unified segmentation framework, key features of foreground objects during the changing process are captured, and the memory features stored in the memory network are dynamically adjusted, thereby improving the accuracy and robustness of foreground object sequence segmentation in various scenarios. Attached Figure Description

[0052] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0053] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a flowchart illustrating an embodiment of the foreground object segmentation method based on contrastive learning in this application.

[0055] Figure 2 This is a flowchart illustrating Embodiment 2 of the foreground object segmentation method based on contrastive learning in this application.

[0056] Figure 3 This is a flowchart illustrating Embodiment 3 of the foreground object segmentation method based on contrastive learning in this application;

[0057] Figure 4 A simplified flowchart illustrating the foreground object segmentation method based on contrastive learning provided in Embodiment 3 of this application;

[0058] Figure 5 This is a schematic diagram of the module structure of the foreground object segmentation device based on contrastive learning according to an embodiment of this application;

[0059] Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the foreground object segmentation method based on contrastive learning in the embodiments of this application.

[0060] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0061] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0062] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0063] The main solution of this application embodiment is as follows: A target segmentation network generates a current prediction mask for the current video frame based on the current video frame features corresponding to the current video frame and the memory features stored in the memory network; a target discrimination network obtains a global feature vector for the current video frame based on the currently extracted features corresponding to the current video frame; similarity is calculated based on the global feature vector and the initial feature vector, and a feature update method is determined based on the vector similarity and a similarity threshold; the memory features are updated according to the feature update method and the current prediction mask to obtain updated memory features; target object segmentation is performed on each video frame of the target video based on the updated memory features until object segmentation of each video frame is completed.

[0064] Segmentation methods based on memory networks and feature matching have become a research hotspot in video foreground object sequence segmentation in recent years. Based on multi-level memory mechanisms and complex matching patterns, some researchers have proposed memory matching models, which have received widespread attention in the field of video foreground object sequence segmentation. STM (Spacetime Memory) uses a non-local matching algorithm to match the features of the current video frame with the features of the target object in the memory network, thereby enhancing the relevant features of the target object in the current video frame. STCN (Space-Time Contextual Network) further improves the matching algorithm and reuses the features encoded by the segmentation network in the memory network to reduce redundant encoding. SWEM (Space-Time Weighted Memory) adopts a multi-stage memory mode, utilizing memory features from different stages to enhance the foreground object sequence segmentation effect of the current video frame. AOT (Adaptive Object Tracking) uses a target re-identification algorithm to complete the synchronous segmentation of multiple foreground objects in the video. DeAOT (Decoupled Adaptive Object Tracking) further decouples feature matching and multi-target differentiation, while using a gated attention matching algorithm to enhance the video foreground object segmentation effect. ISVOS (Instance-Specific Video Object Segmentation) uses instance segmentation algorithms to enhance the segmentation of foreground objects. Cutie (Cut and Track Instance-level Estimation) uses object-level matching algorithms to further reduce the impact of similar interference objects in the same video frame on the segmentation of foreground objects.

[0065] While these segmentation methods based on memory networks and feature matching have achieved good segmentation results, they all use fixed intervals to update the memory network. This makes it difficult to adjust the memory in a timely manner according to the changes of objects in the video, and it is easy to lose key features of objects that appear during video changes, which is crucial for the effective segmentation of targets that have undergone significant changes. At the same time, because they cannot accurately judge the changes of objects, the memory network cannot fuse redundant features when the foreground objects change less, nor can it make targeted enhancements to newly emerging key features when the foreground objects change more, making it difficult to achieve effective segmentation of target objects in complex video scenes.

[0066] This application provides a solution that identifies the changing state of foreground objects within a unified segmentation framework, captures key features of foreground objects during the changing process, and dynamically adjusts the memory features stored in the memory network, thereby improving the accuracy and robustness of video foreground object sequence segmentation in various scenarios.

[0067] It should be noted that the execution subject in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or a foreground object segmentation device based on contrastive learning capable of performing the above functions. The following description uses a foreground object segmentation device based on contrastive learning as the execution subject to illustrate this embodiment and the subsequent embodiments.

[0068] Based on this, embodiments of this application provide a foreground object segmentation method based on contrastive learning, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the foreground object segmentation method based on contrastive learning in this application.

[0069] In this embodiment, the foreground object segmentation method based on contrastive learning includes steps S10 to S50:

[0070] Step S10: The target segmentation network generates the current prediction mask of the current video frame based on the current video frame features corresponding to the current video frame and the memory features stored in the memory network.

[0071] It should be noted that the method framework of this embodiment consists of a target segmentation network, a target discriminator network, and a memory network. The target segmentation network enhances the relevant features of the target object in the current video frame by using the stored feature matching in the target memory network to achieve the segmentation of the target object in the current input video frame; the target discriminator network judges the degree of change of the target object by calculating the feature similarity between different video frames to determine the memory update method in the memory network; the target memory network re-encodes the segmentation result and completes the dynamic update of short-term memory or long-term memory based on the discrimination signal generated by the discriminator network.

[0072] Understandably, for a segmentation network, the training frames are first input into the encoder to obtain the encoded current frame features. These features are then matched with stored features in the memory network to enhance the features related to the target object. The enhanced current frame features are then input into the decoder, and the segmentation network is trained using ground truth masking. Next, test frames are input into the segmentation network to generate a mask, completing the segmentation of the target object. In this embodiment, the trained segmentation network is the target segmentation network. The test frames involved in this embodiment are the video frames used in subsequent practical applications.

[0073] In the specific implementation, for the discriminative network, firstly, training frames are input into the encoder of the discriminative network. The encoded features are then enhanced through a two-stage attention mechanism involving target activation and semantic activation. Projection operations are then performed on the features, and multi-level contrastive learning is used to complete the training process of the discriminative network. Next, test frames are progressively input into the discriminative network to generate feature vectors. By calculating the similarity between the feature vectors of different frames, the update method for the current frame in the memory network is determined. In this embodiment, the trained discriminative network is the target discriminative network.

[0074] It should be noted that for the memory network, firstly, the segmentation results of the segmentation network on the training frames are input into the encoder of the memory network. After fusing the encoded features of the segmentation network on that frame, the enhanced target object features are obtained. Depending on the span of the video frame in the video, the current features are selected to be input into the fusion module of short-term memory or the enhancement module of long-term memory, thus completing the dynamic update of the features stored in the memory network. Then, the segmentation results of the segmentation network on the test frames are input into the memory network. Based on the discrimination signal generated by the trained discriminator network regarding the changes in the target object in the current frame, the short-term memory or long-term memory is dynamically updated to prepare for the segmentation network to match and enhance the segmentation of subsequent frames.

[0075] It is understandable that the target video is the video that needs to be segmented, and the target video contains multiple video frames, with the current video frame being I. t Let t∈{1,2,...,T}, where T represents the number of video frames corresponding to the target video. The current video frame I... t The input is fed into the target segmentation network, which is based on the current video frame I. t Current video frame features The memory features stored in the memory network can be used to generate the current video frame I. t The corresponding current prediction mask Y t .

[0076] In one feasible implementation, step S10 may include steps A11 to A13:

[0077] Step A11: Extract features from the current video frame using the feature extraction network in the target segmentation network to obtain the current video frame features corresponding to the current video frame;

[0078] It should be noted that the current video frame I t The input is fed into the target segmentation network, which uses a feature extraction network to process the current video frame I. t Feature extraction is performed to obtain the features of the current video frame.

[0079]

[0080] Step A12: Perform a matching operation based on the current video frame features and the memory features stored in the memory network to obtain the target enhancement features.

[0081] It should be noted that the current video frame features are matched with the memory features stored in the memory network to obtain the current frame features for target object enhancement, which are called target enhancement features.

[0082]

[0083] Step A13: Input the target enhancement features into the decoder in the target segmentation network to generate the current prediction mask for the current video frame.

[0084] It should be noted that enhancing the target features The input is fed into the decoder in the target segmentation network to generate the current predicted mask Y corresponding to the target object in the current video frame. t .

[0085] In one feasible implementation, step A12 may include steps B121 to B124:

[0086] Step B121: Compress the number of feature channels of the current video frame features to generate a target query.

[0087] Step B122: Determine the feature key and target value based on the memory features stored in the memory network.

[0088] Step B123: Perform similarity calculation based on the target query and the feature key to obtain a similarity matrix.

[0089] Step B124: Generate features based on the similarity matrix and the target value to obtain target enhancement features.

[0090] It should be noted that the target segmentation network compresses the features F of the current video frame through projection operations. t Feature channel data volume to generate target query Determine the stored feature keys based on the memory features stored in the memory network. and the target value stored Target query and feature keys Perform similarity calculations to obtain a similarity matrix. After normalizing the similarity matrix, apply the result to the target value. Weighting is applied to generate current frame features that enhance the target object. Where j∈{1,...,N}, and N represents the number of stored memory frames. The specific calculation process is as follows: Where SoftMax(·) represents the SoftMax operator, and <·,·> represents the L2 similarity calculation process.

[0091] The above is only one possible implementation of step S10 provided in this embodiment. This embodiment does not specifically limit the specific implementation of step S10.

[0092] Step S20: The target discrimination network obtains the global feature vector of the current video frame based on the currently extracted features corresponding to the current video frame.

[0093] It should be noted that the target discrimination network uses a feature extraction network to analyze the current video frame I. t Perform feature extraction to obtain the currently extracted features. This feature is enhanced and projected to obtain the current video frame I. t global feature vector In this embodiment, the global feature vector is used for similarity calculation between subsequent frames and to generate a discrimination signal for the memory network update method.

[0094] Step S30: Calculate the similarity based on the global feature vector and the initial feature vector, and determine the feature update method based on the vector similarity and the similarity threshold.

[0095] It should be noted that the initial feature vector refers to the initial feature vector of the i-th change segment. The global feature vector g t and the initial eigenvector g i A similarity calculation is performed to obtain the vector similarity between the two.

[0096] It is understandable that the vector similarity is compared with the similarity threshold. If the vector similarity is lower than the similarity threshold, it means that the object has changed drastically, and the (i+1)th change segment needs to be activated. At the same time, the long-term memory in the memory network should be updated. In this case, the feature update method is the long-term memory update method. Otherwise, the i-th change segment is maintained, and the short-term memory in the memory network is updated. In this case, the feature update method is the short-term memory update method.

[0097] In one feasible implementation, step S30 may include steps A31 to A34:

[0098] Step A31: Randomly select video frames from multiple video segments corresponding to the target video to obtain multiple extracted frames;

[0099] Step A32: Determine the global feature vector of each extracted frame through the target discrimination network;

[0100] Step A33: Calculate the similarity between adjacent video frames based on the global feature vectors of each extracted frame to obtain multiple target similarities;

[0101] Step A34: Calculate the mean value based on the similarity of each target to obtain the similarity threshold.

[0102] It should be noted that before object segmentation of the target video, the target video is divided into K video segments, and one frame is randomly extracted from each video segment to obtain K video frames. The global feature vector g corresponding to the K video frames is directly calculated using the target discrimination network. Then, the target similarity between the global feature vectors of adjacent video frames is calculated in turn, and the average value of all target similarities is used as the similarity threshold for subsequent discrimination.

[0103] Step S40: Update the memory features according to the feature update method and the current prediction mask to obtain the updated memory features;

[0104] It should be noted that the dynamic update of the memory network storage process is mainly achieved by the similarity fusion module and the change highlighting module. When the target discrimination network does not require the activation of new change segments, the feature update method is the short-term memory update method. The similarity fusion module encodes the features corresponding to the current prediction mask as follows: memory network encoded feature M t Fusion is performed to eliminate redundant features in memory and update short-term memories in the current memory features. This yields the updated memory features; when the discriminant network generates a signal that initiates a new change segment, the feature update method becomes long-term memory update, and the change highlighting module compares the current encoded features M... t The initial memory network encoding features M i Highlighting the characteristics of change and using them as long-term memory features of the current change segment. This results in updated memory features.

[0105] Step S50: Perform target object segmentation on each video frame of the target video according to the updated memory features until object segmentation of each video frame is completed.

[0106] It should be noted that the updated memory features participate in the target segmentation network's mask prediction process for the next video frame. The target discriminator network determines the feature update method for the memory network based on the global feature vector. The memory network updates the currently stored memory features according to the feature update method, repeating all the above steps until the target object is segmented in all video frames of the target video. In this embodiment, the memory features corresponding to the first video frame are obtained through annotation. Subsequent memory features are updated and stored based on the previous video frame and the corresponding update method.

[0107] This embodiment generates a current prediction mask for the current video frame using a target segmentation network based on the current video frame features and memory features stored in a memory network. A target discrimination network obtains a global feature vector for the current video frame based on the currently extracted features. Similarity is calculated between the global feature vector and an initial feature vector, and a feature update method is determined based on vector similarity and a similarity threshold. The memory features are updated according to the feature update method and the current prediction mask to obtain updated memory features. Target object segmentation is performed on each video frame of the target video using the updated memory features until object segmentation of each video frame is complete. Through this method, the changing state of foreground objects is determined within a unified segmentation framework, key features of foreground objects during the changing process are captured, and the memory features stored in the memory network are dynamically adjusted, thereby improving the accuracy and robustness of foreground object sequence segmentation in various scenarios.

[0108] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 The foreground object segmentation method based on contrastive learning further includes steps S21 to S25 in step S20:

[0109] Step S21: Extract features from the current video frame using the feature extraction network in the target discrimination network to obtain the current extracted features corresponding to the current video frame.

[0110] It should be noted that the target discrimination network uses a feature extraction network to analyze the current video frame I. t Perform feature extraction to obtain the currently extracted features.

[0111] Step S22: Map the currently extracted features to obtain the first query, the first key, and the first value.

[0112] It should be noted that the target discrimination network obtains the currently extracted feature X. t Then, a two-stage attention mechanism operation, involving target activation and semantic activation, is performed to obtain the global feature vector g of the current video frame. t .

[0113] Understandably, when performing target activation, the target discrimination network first processes the currently extracted feature X. t Mapping yields the first query First key

[0114] Step S23: Obtain the target activation feature based on the first query, the first key, and the first value.

[0115] It should be noted that the target discrimination network performs similarity calculations on the first query and the first key, and after normalizing the similarity matrix, it weights the first value to obtain the target activated features, thus obtaining the target activated features. The specific calculation process is as follows:

[0116]

[0117] Step S23: Perform semantic activation based on the second query, second key, and second value corresponding to the target activation feature to obtain the semantic activation feature.

[0118] It should be noted that the target activation feature X obtained after activating the target... t,s The second query is obtained after further mapping. Second key and the second value The similarity is calculated between the second query and the second key. After normalizing the similarity matrix, the second value is weighted to obtain the target activated features, thus yielding semantic activation features. The specific calculation process is as follows:

[0119] Step S24: Obtain the global feature vector of the current video frame based on the semantic activation features.

[0120] It should be noted that semantic activation feature X t,sc After passing through a global average pooling layer and a multivariate perceptron, the global feature vector g of the current video frame is obtained. t .

[0121] Understandably, in order to make the target discrimination network ultimately generate the global feature vector g t To better represent the symbolic information of the target object, the training process employs frame-level, target-level, and pixel-level contrastive learning loss functions to supervise the features. The specific process is as follows: Frame-level contrastive learning loss function: The encoder parameters of the discriminator network are initialized using the encoder parameters of the pre-trained segmentation network. During training, the input video is divided into K equal segments, and one frame is randomly sampled from each segment, for a total of K frames. Then, one frame is randomly sampled from each of the other K videos, for a total of K frames. A training batch containing 2K video frames is considered. For the K frames randomly sampled from the same video, the encoder of the discriminator network is first used to encode them to obtain features. To further balance the semantic contributions of the target object and the background, the number of target object pixels n in each frame is calculated using a ground truth mask. O and the number of background pixels nB Then, K global vectors are obtained through a layer of global weighted pooling. Then each global vector is treated as a target vector. Calculate the similarity between the target vector and the other K-1 vectors. The vector with the highest similarity is considered a positive sample pair, and the vector with the lowest similarity is considered a negative sample pair. The overall calculation process can be expressed by the following formula: Where O and B represent the target object and background region, respectively, and each target video frame I α and I respectively β I γ Construct positive and negative sample pairs. Then I γ Together with K frames randomly sampled from K other videos, a negative sample library N is formed. After determining the positive and negative samples corresponding to each video frame, features are obtained by encoding using target activation and semantic activation. Enhancement is performed and the features are ultimately mapped to a global feature vector g. t The frame-level comparative learning supervision is as follows: τ is used as a hyperparameter to control the training convergence rate.

[0122] In this embodiment, the target-level contrastive learning loss function is used to further enhance the semantic attention of the global feature vector to the target object after clarifying the positive and negative sample relationship between video frames. The specific process is as follows: based on the target video frame I... α and its corresponding positive sample I β And in the negative sample library N, the region O where the target object is located is sequentially subjected to local region average pooling of its encoded feature X, to obtain the target level vector o, o + and o - ∈N o The target-level comparative learning supervision for them is as follows: Pixel-level contrastive learning loss function: Since video object segmentation is a pixel-dense task, it requires more specific pixel-level supervised learning. For target video frame I... α A pixel-level feature vector p needs to be represented in the positive and negative sample pairs I. β and I γ Find the pixel-level feature vector that is most similar to it. The calculation process is as follows: Where p s That is, the positive sample p of the feature vector p. + At the same time, I β and I γ The remaining pixels p - Together they form the negative sample library N p Pixel-level contrastive learning supervision is performed on them as follows:

[0123] This embodiment extracts features from the current video frame using a feature extraction network within a target discrimination network, obtaining the currently extracted features corresponding to the current video frame. The currently extracted features are then mapped to obtain a first query, a first key, and a first value. A target activation feature is obtained based on the first query, the first key, and the first value. Semantic activation is performed based on the second query, the second key, and the second value corresponding to the target activation feature, obtaining a semantic activation feature. Finally, the global feature vector of the current video frame is obtained based on the semantic activation feature. This method accurately obtains the global feature vector of the current video frame, laying the foundation for subsequent updates to the memory features.

[0124] Based on the first embodiment of this application, in the third embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 Step S40 in the contrastive learning-based foreground object segmentation method further includes steps S41 to S45:

[0125] Step S41: When the feature update method is short-term memory update method, obtain the initial encoded features of the segment.

[0126] It should be noted that the dynamic updating of the memory network storage process is mainly achieved by the similarity fusion module and the change highlighting module. When the discrimination network generation does not require opening new change segments, the similarity fusion module encodes feature M in the memory network. t Fusion is performed to eliminate redundant features in memory and update current short-term memory. When the discrimination network generates a signal that enables a new change segment, the change highlighting module compares the current encoded features M. t The initial memory network encoding features M i Highlighting the characteristics of change and using them as long-term memory features of the current change segment.

[0127] It is understandable that when the feature update method is short-term memory update, it means that the short-term memory of the memory features stored in the memory network needs to be updated. At this time, the initial encoded features M of the changed segment are obtained. i That is, the initial coding features of the fragment.

[0128] Step S42: Determine the current long-term memory based on the memory features, and obtain the current coding features based on the current prediction mask and the current video frame features.

[0129] It should be noted that the current long-term memory is determined based on the memory features stored in the memory network, while the memory network uses a feature extraction network to predict the current mask Y. t Re-encode and fuse current video frame features F t The memory coding features of the current video frame are obtained. In this embodiment, the current coding feature refers to the memory coding feature M of the current video frame. t .

[0130] Step S43: Perform feature concatenation based on the current encoding features and the initial encoding features of the segment to obtain concatenated features;

[0131] Step S44: Perform multi-scale fusion on the spliced ​​features to obtain the target short-term memory;

[0132] Step S45: Obtain updated memory features based on the target short-term memory and the current long-term memory.

[0133] It should be noted that the similarity fusion module in the memory network will encode the current feature M. t and fragment initial coding features M i The features are concatenated (Cat) to obtain concatenated features. Then, multi-scale fusion is performed on these concatenated features. Multi-scale fusion comprises four branches, each containing an average pooling layer (AP) and a deformation convolutional layer (DC), allowing the features to focus more on the deformation information of the object. The overall fusion process can be expressed by the following formula: M j =DC j (AP j (Cat(M t M i The outputs of the four branches are concatenated and fused using a 3×3 depthwise separable convolution to generate the final short-term memory. Based on current long-term memory and target short-term memory Together they form the updated memory features.

[0134] In one feasible implementation, step S40 may further include steps A41 to A45:

[0135] Step A41: When the feature update method is long-term memory update method, determine the current short-term memory based on the memory features, and obtain the current encoded features based on the current prediction mask and the current video frame features;

[0136] When the feature update method is long-term memory update, it means that the long-term memory in the memory features stored in the memory network needs to be updated. At this time, the current short-term memory is determined based on the memory features stored in the memory network, and the memory network uses a feature extraction network to update the current prediction mask Y. t Re-encode and fuse current video frame features F t The memory coding features M of the current video frame are obtained. t .

[0137] Step A42: Calculate the similarity based on the current coding features and the initial coding features of the segment to obtain a similarity matrix;

[0138] Step A43: Perform feature processing based on the similarity matrix and the initial encoded features of the segment to determine the adjustment weights;

[0139] Step A44: Perform feature processing on the current encoded features according to the adjusted weights to obtain the target long-term memory;

[0140] Step A45: Obtain updated memory features based on the target long-term memory and the current short-term memory.

[0141] It should be noted that, in order to further highlight and amplify the changes in the target object features at the beginning and end of a change segment, the change salience module in the memory network first processes the currently encoded feature M. t and fragment initial coding features M i Similarity calculations are performed to obtain a similarity matrix. After normalizing the similarity matrix, the initial encoding features M for the fragments are then determined. i The weighted average process can be represented as follows: in This indicates the current encoded feature M t The transpose of the first feature is then used. The features and feature differences are then concatenated, and a 1×1 convolutional layer (Conv) is used to reduce the number of channels in the concatenated feature. This feature is then normalized, and the result is the adjusted weights. The current encoded feature M is adjusted by adjusting the weights. t This is done to enhance the changing characteristics of the target object. The specific process can be represented as follows: in Represents element-wise product. This will serve as the long-term memory corresponding to the current change segment, i.e., the target long-term memory. Based on current short-term memory and target long-term memory Together they form the updated memory features.

[0142] In this embodiment, when the feature update method is long-term memory update, the current short-term memory is determined based on the memory features, and the current encoded features are obtained based on the current prediction mask and the current video frame features. A similarity matrix is ​​obtained by calculating the similarity between the current encoded features and the initial encoded features of the segment. Feature processing is performed based on the similarity matrix and the initial encoded features of the segment to determine adjustment weights. Feature processing is then performed on the current encoded features based on the adjustment weights to obtain the target long-term memory. Updated memory features are obtained based on the target long-term memory and the current short-term memory. Through this method, new long-term memories are generated by highlighting differences, thereby completing the dynamic update of the memory stored in the memory network, which can effectively improve the segmentation effect of target objects in subsequent frames.

[0143] This embodiment obtains the initial encoding features of a segment when the feature update method is short-term memory update; determines the current long-term memory based on the memory features, and obtains the current encoding features based on the current prediction mask and the current video frame features; concatenates the current encoding features and the initial encoding features of the segment to obtain concatenated features; performs multi-scale fusion on the concatenated features to obtain the target short-term memory; and obtains the updated memory features based on the target short-term memory and the current long-term memory. Through the above method, new short-term memories are generated by similarity fusion, thereby completing the dynamic update of the memory stored in the memory network, which can effectively improve the segmentation effect of target objects in subsequent frames.

[0144] For example, to help understand the implementation process of the foreground object segmentation method based on contrastive learning obtained by combining this embodiment with the above embodiment one, please refer to... Figure 4 , Figure 4 A simplified flowchart of a foreground object segmentation method based on contrastive learning is provided, specifically:

[0145] The foreground object segmentation method based on contrastive learning in this embodiment consists of a segmentation network, a discriminant network, and a memory network. S1: The segmentation network enhances the relevant features of the target object in the current video frame by using stored feature matching in the memory network, thereby segmenting the target object in the current input video frame. The discriminant network determines the degree of change of the target object by calculating the feature similarity between different video frames, thus deciding the memory update method in the memory network. The memory network re-encodes the segmentation result and, based on the discriminant signal generated by the discriminant network, dynamically updates the short-term or long-term memory.

[0146] S2. For the segmentation network, firstly, the training frames are input into the encoder of the segmentation network to obtain the encoded current frame features. The stored features in the memory network are used to match and enhance the features related to the target object. The enhanced current frame features are input into the decoder of the segmentation network, and the segmentation network is trained using mask ground truth supervision. Then, the test frames are input into the segmentation network to generate a mask, and the target object is segmented.

[0147] S3. For the discriminative network, firstly, the training frames are input into the encoder of the discriminative network. The encoded features are enhanced by the two-stage attention mechanism of target activation and semantic activation. Then, the features are projected and multi-level contrastive learning is used to complete the training process of the discriminative network. Then, the test frames are input into the discriminative network step by step to generate feature vectors. By calculating the similarity between the feature vectors of different frames, the update method of the current frame in the memory network is determined.

[0148] S4. For the memory network, firstly, the segmentation results of the segmentation network on the training frames are input into the encoder of the memory network. After fusing the encoded features of the segmentation network on the frame, the enhanced target object features are obtained. Based on the span of the video frame in the video, the current features are selected to be input into the fusion module of short-term memory or the enhancement module of long-term memory to complete the dynamic update of the features stored in the memory network. Then, the segmentation results of the segmentation network on the test frames are input into the memory network. Based on the discrimination signal generated by the trained discriminator network for the changes of the target object in the current frame, the short-term memory or long-term memory is dynamically updated to prepare for the segmentation network to match and enhance the segmentation of subsequent frames.

[0149] This embodiment is based on a contrastive learning adaptive memory update framework to keenly perceive the degree of change of the target object, accurately capture the key features of the target object in the process of change, and dynamically adjust the stored features in the memory network, thereby improving the accuracy of target object segmentation in complex scenes and showing robust segmentation performance in dealing with various complex scenes.

[0150] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the foreground object segmentation method based on contrast learning in this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0151] This application also provides a foreground object segmentation device based on contrastive learning, please refer to... Figure 5 The foreground object segmentation device based on contrastive learning includes:

[0152] The generation module 10 is used to generate the current prediction mask of the current video frame through the target segmentation network based on the current video frame features corresponding to the current video frame and the memory features stored in the memory network.

[0153] Processing module 20 is used to obtain the global feature vector of the current video frame based on the currently extracted features corresponding to the current video frame through a target discrimination network;

[0154] The calculation module 30 is used to calculate the similarity based on the global feature vector and the initial feature vector, and to determine the feature update method based on the vector similarity and the similarity threshold.

[0155] Update module 40 is used to update the memory features according to the feature update method and the current prediction mask to obtain the updated memory features;

[0156] The segmentation module 50 is used to segment the target objects in each video frame of the target video according to the updated memory features until the object segmentation of each video frame is completed.

[0157] Optionally, the generation module 10 is further configured to:

[0158] The current video frame features are obtained by extracting features from the feature extraction network in the target segmentation network.

[0159] The target enhancement features are obtained by performing a matching operation based on the current video frame features and the memory features stored in the memory network;

[0160] The target enhancement features are input into the decoder in the target segmentation network to generate the current prediction mask for the current video frame.

[0161] Optionally, the generation module 10 is further configured to:

[0162] The number of feature channels of the current video frame features is compressed to generate a target query;

[0163] The feature keys and target values ​​are determined based on the memory features stored in the memory network;

[0164] A similarity matrix is ​​obtained by performing a similarity calculation based on the target query and the feature key;

[0165] Feature generation is performed based on the similarity matrix and the target value to obtain target enhancement features.

[0166] Optionally, the processing module 20 is further configured to:

[0167] The feature extraction network in the target discrimination network is used to extract features from the current video frame to obtain the current extracted features corresponding to the current video frame.

[0168] The extracted features are mapped to obtain a first query, a first key, and a first value;

[0169] The target activation feature is obtained based on the first query, the first key, and the first value;

[0170] Semantic activation is performed based on the second query, second key, and second value corresponding to the target activation feature to obtain the semantic activation feature;

[0171] The global feature vector of the current video frame is obtained based on the semantic activation features.

[0172] Optionally, the computing module 30 is further configured to:

[0173] Multiple extracted frames are obtained by randomly selecting video frames from multiple video segments corresponding to the target video.

[0174] The global feature vector of each extracted frame is determined by the target discrimination network.

[0175] The similarity between adjacent video frames is calculated based on the global feature vector of each extracted frame to obtain multiple target similarities.

[0176] The similarity threshold is obtained by calculating the mean of the similarity of each target.

[0177] Optionally, the update module 40 is further configured to:

[0178] When the feature update method is short-term memory update, the initial encoded features of the segment are obtained;

[0179] The current long-term memory is determined based on the memory features, and the current coding features are obtained based on the current prediction mask and the current video frame features;

[0180] Based on the current encoded features and the initial encoded features of the segment, feature concatenation is performed to obtain concatenated features;

[0181] Multi-scale fusion of the spliced ​​features yields the target short-term memory;

[0182] The updated memory features are obtained based on the target short-term memory and the current long-term memory.

[0183] Optionally, the update module 40 is further configured to:

[0184] When the feature update method is long-term memory update method, the current short-term memory is determined according to the memory features, and the current coding features are obtained according to the current prediction mask and the current video frame features;

[0185] A similarity matrix is ​​obtained by calculating the similarity based on the current encoding features and the initial encoding features of the segment;

[0186] Feature processing is performed based on the similarity matrix and the initial encoded features of the segment to determine the adjustment weights;

[0187] Based on the adjusted weights, feature processing is performed on the current encoded features to obtain the target long-term memory;

[0188] The updated memory features are obtained based on the target long-term memory and the current short-term memory.

[0189] The foreground object segmentation apparatus based on contrastive learning provided in this application employs the foreground object segmentation method based on contrastive learning in the above embodiments, and can solve the technical problem of how to improve the accuracy and robustness of video foreground object sequence segmentation. Compared with the prior art, the beneficial effects of the foreground object segmentation apparatus based on contrastive learning provided in this application are the same as those of the foreground object segmentation method based on contrastive learning provided in the above embodiments, and other technical features in the foreground object segmentation apparatus based on contrastive learning are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0190] This application provides a foreground object segmentation device based on contrastive learning. The foreground object segmentation device based on contrastive learning includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the foreground object segmentation method based on contrastive learning in the first embodiment described above.

[0191] The following is for reference. Figure 6 This document illustrates a structural schematic diagram of a foreground object segmentation device based on contrastive learning, suitable for implementing embodiments of this application. The foreground object segmentation device based on contrastive learning in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The foreground object segmentation device based on contrastive learning shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0192] like Figure 6As shown, the foreground object segmentation device based on contrastive learning may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the contrastive learning-based foreground object segmentation device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the contrast-learning-based foreground object segmentation device to communicate wirelessly or wiredly with other devices to exchange data. Although contrast-learning-based foreground object segmentation devices with various systems are shown in the figures, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems can be implemented alternatively.

[0193] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0194] The foreground object segmentation device based on contrastive learning provided in this application, employing the foreground object segmentation method based on contrastive learning in the above embodiments, can solve the technical problem of how to improve the accuracy and robustness of video foreground object sequence segmentation. Compared with the prior art, the beneficial effects of the foreground object segmentation device based on contrastive learning provided in this application are the same as those of the foreground object segmentation method based on contrastive learning provided in the above embodiments, and other technical features in this foreground object segmentation device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0195] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0196] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0197] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the foreground object segmentation method based on contrastive learning in the above embodiments.

[0198] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0199] The aforementioned computer-readable storage medium may be included in a foreground object segmentation device based on contrastive learning; or it may exist independently and not assembled into a foreground object segmentation device based on contrastive learning.

[0200] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by a foreground object segmentation device based on contrastive learning, the foreground object segmentation device based on contrastive learning performs the following actions: 1) Generates a current prediction mask for the current video frame using a target segmentation network based on the current video frame features corresponding to the current video frame and memory features stored in a memory network; 2) Obtains a global feature vector for the current video frame using a target discrimination network based on the currently extracted features corresponding to the current video frame; 3) Calculates similarity between the global feature vector and an initial feature vector, and determines a feature update method based on the vector similarity and a similarity threshold; 4) Updates the memory features according to the feature update method and the current prediction mask to obtain updated memory features; 5) Segments target objects in each video frame of the target video according to the updated memory features until object segmentation in each video frame is completed.

[0201] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0202] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0203] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0204] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described contrastive learning-based foreground object segmentation method, thereby solving the technical problem of how to improve the accuracy and robustness of video foreground object sequence segmentation. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the contrastive learning-based foreground object segmentation method provided in the above embodiments, and will not be repeated here.

[0205] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the foreground object segmentation method based on contrastive learning as described above.

[0206] The computer program product provided in this application can solve the technical problem of how to improve the accuracy and robustness of video foreground object sequence segmentation. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the foreground object segmentation method based on contrastive learning provided in the above embodiments, and will not be repeated here.

[0207] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A foreground object segmentation method based on contrastive learning, characterized in that, The foreground object segmentation method based on contrastive learning includes: The target segmentation network generates the current prediction mask for the current video frame based on the current video frame features corresponding to the current video frame and the memory features stored in the memory network. The target discrimination network obtains the global feature vector of the current video frame based on the currently extracted features corresponding to the current video frame; Similarity is calculated based on the global feature vector and the initial feature vector, and the feature update method is determined based on the vector similarity and the similarity threshold. The initial feature vector is the initial feature vector of the changed segment in the memory network. The memory features are updated according to the feature update method and the current prediction mask to obtain the updated memory features; Based on the updated memory features, target object segmentation is performed on each video frame of the target video until object segmentation of each video frame is completed.

2. The method as described in claim 1, characterized in that, The step of generating the current prediction mask for the current video frame using the target segmentation network based on the current video frame features corresponding to the current video frame and the memory features stored in the memory network includes: The current video frame features are obtained by extracting features from the feature extraction network in the target segmentation network. The target enhancement features are obtained by performing a matching operation based on the current video frame features and the memory features stored in the memory network; The target enhancement features are input into the decoder in the target segmentation network to generate the current prediction mask for the current video frame.

3. The method as described in claim 2, characterized in that, The step of performing a matching operation based on the current video frame features and the memory features stored in the memory network to obtain the target enhancement features includes: The number of feature channels of the current video frame features is compressed to generate a target query; The feature keys and target values ​​are determined based on the memory features stored in the memory network; A similarity matrix is ​​obtained by performing a similarity calculation based on the target query and the feature key; Feature generation is performed based on the similarity matrix and the target value to obtain target enhancement features.

4. The method as described in claim 1, characterized in that, The step of obtaining the global feature vector of the current video frame through the target discrimination network based on the currently extracted features corresponding to the current video frame includes: The feature extraction network in the target discrimination network is used to extract features from the current video frame to obtain the current extracted features corresponding to the current video frame. The currently extracted features are mapped to obtain a first query, a first key, and a first value; The target activation feature is obtained based on the first query, the first key, and the first value; Semantic activation is performed based on the second query, second key, and second value corresponding to the target activation feature to obtain the semantic activation feature. The second query, second key, and second value are mapped from the target activation feature. The global feature vector of the current video frame is obtained based on the semantic activation features.

5. The method as described in claim 1, characterized in that, Before the step of calculating similarity based on the global feature vector and the initial feature vector, and determining the feature update method based on the vector similarity and the similarity threshold, the method further includes: Multiple extracted frames are obtained by randomly selecting video frames from multiple video segments corresponding to the target video. The global feature vector of each extracted frame is determined by the target discrimination network. The similarity between adjacent video frames is calculated based on the global feature vector of each extracted frame to obtain multiple target similarities. The similarity threshold is obtained by calculating the mean of the similarity of each target.

6. The method as described in claim 1, characterized in that, The step of updating the memory features according to the feature update method and the current prediction mask to obtain the updated memory features includes: When the feature update method is short-term memory update method, the initial encoding features of the segment are obtained, and the initial encoding features of the segment are the initial encoding features of the changed segments in the memory network; The current long-term memory is determined based on the memory features, and the current coding features are obtained based on the current prediction mask and the current video frame features; Based on the current encoding features and the initial encoding features of the segment, feature concatenation is performed to obtain concatenated features; Multi-scale fusion of the spliced ​​features yields the target short-term memory; The updated memory features are obtained based on the target short-term memory and the current long-term memory.

7. The method as described in claim 1, characterized in that, The step of updating the memory features according to the feature update method and the current prediction mask to obtain the updated memory features includes: When the feature update method is long-term memory update method, the current short-term memory is determined according to the memory features, and the current coding features are obtained according to the current prediction mask and the current video frame features; A similarity matrix is ​​obtained by calculating the similarity based on the current encoding features and the initial encoding features of the segment; Feature processing is performed based on the similarity matrix and the initial encoded features of the segment to determine the adjustment weights; Based on the adjusted weights, feature processing is performed on the current encoded features to obtain the target long-term memory; The updated memory features are obtained based on the target long-term memory and the current short-term memory.

8. A foreground object segmentation device based on contrastive learning, characterized in that, The foreground object segmentation device based on contrastive learning includes: The generation module is used to generate the current prediction mask of the current video frame through the target segmentation network based on the current video frame features corresponding to the current video frame and the memory features stored in the memory network. The processing module is used to obtain the global feature vector of the current video frame based on the currently extracted features corresponding to the current video frame through the target discrimination network; The calculation module is used to calculate the similarity based on the global feature vector and the initial feature vector, and to determine the feature update method based on the vector similarity and the similarity threshold. The initial feature vector is the initial feature vector of the changed segment in the memory network. The update module is used to update the memory features according to the feature update method and the current prediction mask to obtain the updated memory features; The segmentation module is used to segment the target objects in each video frame of the target video according to the updated memory features until the object segmentation of each video frame is completed.

9. A foreground object segmentation device based on contrastive learning, characterized in that, The foreground object segmentation device based on contrastive learning includes: a memory, a processor, and a foreground object segmentation program based on contrastive learning stored in the memory and executable on the processor, wherein the foreground object segmentation program based on contrastive learning is configured to implement the foreground object segmentation method based on contrastive learning as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a foreground object segmentation program based on contrastive learning, which, when executed by a processor, implements the foreground object segmentation method based on contrastive learning as described in any one of claims 1 to 7.