Contrast learning-based foreground object segmentation method, apparatus and device, and storage medium

By generating predictive masks and global feature vectors for video frames through target segmentation and discriminant networks, and dynamically updating memory features, the problem of difficulty in adjusting memory networks in existing technologies is solved, thereby improving the accuracy and robustness of video foreground object sequence segmentation.

CN119006826AActive Publication Date: 2024-11-22PENG CHENG LAB
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411152736.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-21
Publication Date
2024-11-22
Estimated Expiration
2044-08-21

AI Technical Summary

Technical Problem

Existing video foreground object sequence segmentation methods based on memory networks and feature matching struggle to adjust the memory in a timely manner according to the changing characteristics of objects in the video. This leads to the loss of key features, making it impossible to accurately judge the changes in objects, and to fuse redundant features or enhance newly emerging key features in a timely manner. Consequently, it is difficult to achieve effective segmentation of target objects in complex video scenes.

Method used

The target segmentation network and target discrimination network generate the prediction mask and global feature vector of the current video frame. The feature update method is determined by similarity calculation and similarity threshold. The memory features in the memory network are dynamically adjusted to realize the update of short-term or long-term memory, thereby improving the accuracy and robustness of video foreground object sequence segmentation.

Benefits of technology

By identifying the changing states of foreground objects within a unified segmentation framework, capturing key features, and dynamically adjusting the memory features stored in the memory network, the accuracy and robustness of video foreground object sequence segmentation are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006826A_ABST
    Figure CN119006826A_ABST
Patent Text Reader

Abstract

The invention discloses a foreground object segmentation method and device based on comparative learning, equipment and a storage medium, and relates to the technical field of video processing, and the method comprises the steps: generating a current prediction mask of a current video frame through a target segmentation network; obtaining a global feature vector of the current video frame through a target discrimination network; determining a feature updating mode according to the global feature vector and the initial feature vector in combination with a similarity threshold value; and updating the memory feature according to the feature updating mode and the current prediction mask, and performing target object segmentation on each subsequent video frame in the target video according to the updated memory feature until object segmentation of each video frame is completed. Through the mode, the change state of the foreground object is judged in the unified segmentation framework, the key features of the foreground object in the change process are captured, and the memory features stored in the memory network are dynamically adjusted, so that the accuracy and robustness of video foreground object sequence segmentation in various scenes are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a foreground object segmentation method, device, equipment and storage medium based on contrastive learning. Background Art

[0002] In recent years, segmentation methods based on memory networks and feature matching have become one of the research hotspots in the segmentation of video foreground object sequences. Based on multi-level memory mechanisms and complex matching patterns, some researchers have proposed memory matching models, which have received widespread attention in the field of video foreground object sequence segmentation. However, the current segmentation methods based on memory networks and feature matching all use fixed intervals to update the memory network, which makes it difficult to fully adjust the memory in a timely manner based on the changing characteristics of the object itself in the video, and it is easy to lose the key features of the object that appear during the video change process, which is crucial for the effective segmentation of targets that have undergone severe changes. At the same time, due to the inability to accurately judge the changes of objects, the memory network cannot timely integrate the redundant features of foreground objects with small changes, and cannot make targeted enhancements to the key features that appear when the foreground objects change greatly, making it difficult to complete the effective segmentation of target objects in complex scene videos. Summary of the invention

[0003] The main purpose of this application is to provide a foreground object segmentation method, device, equipment and storage medium based on contrastive learning, aiming to solve the technical problem of how to improve the accuracy and robustness of video foreground object sequence segmentation.

[0004] To achieve the above objectives, the present application proposes a foreground object segmentation method based on contrastive learning, the foreground object segmentation method based on contrastive learning comprising:

[0005] Generate a current prediction mask of the current video frame according to the current video frame features corresponding to the current video frame and the memory features stored in the memory network through the target segmentation network;

[0006] Obtaining a global feature vector of the current video frame according to the current extracted features corresponding to the current video frame through a target discrimination network;

[0007] Calculating similarity based on the global feature vector and the initial feature vector, and determining a feature update method based on the vector similarity and a similarity threshold;

[0008] updating the memory feature according to the feature update mode and the current prediction mask to obtain an updated memory feature;

[0009] Target object segmentation is performed on each video frame of the target video according to the updated memory features until the object segmentation of each video frame is completed.

[0010] In one embodiment, the step of generating a current prediction mask of the current video frame according to the current video frame features corresponding to the current video frame and the memory features stored in the memory network through the target segmentation network includes:

[0011] Extract features of the current video frame through a feature extraction network in the target segmentation network to obtain current video frame features corresponding to the current video frame;

[0012] Performing a matching operation based on the current video frame feature and the memory feature stored in the memory network to obtain a target enhancement feature;

[0013] The target enhancement feature is input into a decoder in the target segmentation network to generate a current prediction mask for a current video frame.

[0014] In one embodiment, the step of performing a matching operation based on the current video frame feature and the memory feature stored in the memory network to obtain the target enhancement feature includes:

[0015] Compressing the number of feature channels of the current video frame feature to generate a target query;

[0016] Determine feature keys and target values ​​according to memory features stored in the memory network;

[0017] Perform similarity calculation based on the target query and the feature key to obtain a similarity matrix;

[0018] Features are generated according to the similarity matrix and the target value to obtain target enhanced features.

[0019] In one embodiment, the step of obtaining the global feature vector of the current video frame according to the current extracted features corresponding to the current video frame through the target discrimination network includes:

[0020] Extract features of the current video frame through a feature extraction network in the target discrimination network to obtain current extracted features corresponding to the current video frame;

[0021] Mapping the currently extracted feature to obtain a first query, a first key, and a first value;

[0022] Obtain a target activation feature according to the first query, the first key, and the first value;

[0023] Performing semantic activation according to a second query, a second key, and a second value corresponding to the target activation feature to obtain a semantic activation feature;

[0024] A global feature vector of the current video frame is obtained according to the semantic activation feature.

[0025] In one embodiment, before the step of calculating similarity based on the global feature vector and the initial feature vector and determining the feature update method based on the vector similarity and the similarity threshold, the method further includes:

[0026] Randomly selecting video frames from multiple video clips corresponding to the target video to obtain multiple extracted frames;

[0027] Determining the global feature vector of each extracted frame through the target discrimination network;

[0028] The similarity between adjacent video frames is calculated based on the global feature vector of each extracted frame to obtain multiple target similarities;

[0029] The similarity threshold is obtained by calculating the mean value of each target similarity.

[0030] In one embodiment, the step of updating the memory feature according to the feature update mode and the current prediction mask to obtain the updated memory feature includes:

[0031] When the feature updating mode is a short-term memory updating mode, obtaining the initial encoding feature of the fragment;

[0032] Determining a current long-term memory according to the memory feature, and obtaining a current encoding feature according to the current prediction mask and the current video frame feature;

[0033] Perform feature splicing according to the current coding feature and the initial coding feature of the fragment to obtain a splicing feature;

[0034] Performing multi-scale fusion on the splicing features to obtain target short-term memory;

[0035] An updated memory feature is obtained according to the target short-term memory and the current long-term memory.

[0036] In one embodiment, the step of updating the memory feature according to the feature update mode and the current prediction mask to obtain the updated memory feature includes:

[0037] When the feature update mode is a long-term memory update mode, determining the current short-term memory according to the memory feature, and obtaining the current encoding feature according to the current prediction mask and the current video frame feature;

[0038] Calculate similarity based on the current encoding feature and the initial encoding feature of the segment to obtain a similarity matrix;

[0039] Perform feature processing according to the similarity matrix and the initial encoding features of the fragment to determine an adjustment weight;

[0040] Performing feature processing on the current encoding feature according to the adjustment weight to obtain a target long-term memory;

[0041] An updated memory feature is obtained according to the target long-term memory and the current short-term memory.

[0042] In addition, to achieve the above purpose, the present application also proposes a foreground object segmentation device based on contrastive learning, and the foreground object segmentation device based on contrastive learning includes:

[0043] A generation module, configured to generate a current prediction mask of the current video frame according to a current video frame feature corresponding to the current video frame and a memory feature stored in the memory network through a target segmentation network;

[0044] A processing module, configured to obtain a global feature vector of the current video frame according to a current extracted feature corresponding to the current video frame through a target discrimination network;

[0045] A calculation module, used for performing similarity calculation based on the global feature vector and the initial feature vector, and determining a feature update method based on the vector similarity and a similarity threshold;

[0046] An updating module, used for updating the memory feature according to the feature updating mode and the current prediction mask to obtain an updated memory feature;

[0047] The segmentation module is used to segment the target object in each video frame of the target video according to the updated memory features until the object segmentation of each video frame is completed.

[0048] In addition, to achieve the above-mentioned purpose, the present application also proposes a foreground object segmentation device based on contrastive learning, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the foreground object segmentation method based on contrastive learning as described above.

[0049] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the foreground object segmentation method based on contrast learning as described above are implemented.

[0050] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the foreground object segmentation method based on contrast learning as described above are implemented.

[0051] The present application generates a current prediction mask of the current video frame according to the current video frame features corresponding to the current video frame and the memory features stored in the memory network through the target segmentation network; obtains the global feature vector of the current video frame according to the current extracted features corresponding to the current video frame through the target discrimination network; performs similarity calculation based on the global feature vector and the initial feature vector, and determines the feature update method based on the vector similarity and the similarity threshold; updates the memory features according to the feature update method and the current prediction mask to obtain the updated memory features; performs target object segmentation on each video frame of the target video according to the updated memory features until the object segmentation of each video frame is completed. Through the above method, the change state of the foreground object is judged within a unified segmentation framework, the key features of the foreground object in the change process are captured, and the memory features stored in the memory network are dynamically adjusted, thereby improving the accuracy and robustness of the segmentation of the video foreground object sequence in various scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0054] Figure 1 A flowchart diagram of a first embodiment of a foreground object segmentation method based on contrast learning of the present application;

[0055] Figure 2 A flowchart diagram of a second embodiment of a foreground object segmentation method based on contrast learning of the present application;

[0056] Figure 3 A flowchart diagram of a third embodiment of the foreground object segmentation method based on contrast learning of the present application;

[0057] Figure 4 A schematic diagram of a simplified process of a foreground object segmentation method based on contrastive learning provided in Example 3 of the present application;

[0058] Figure 5 This is a schematic diagram of the module structure of a foreground object segmentation device based on contrast learning in an embodiment of the present application;

[0059] Figure 6 Schematic diagram of the device structure of the hardware operating environment involved in the foreground object segmentation method based on contrast learning in the embodiment of the present application.

[0060] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0061] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0062] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0063] The main solution of the embodiment of the present application is: generate a current prediction mask of the current video frame according to the current video frame features corresponding to the current video frame and the memory features stored in the memory network through the target segmentation network; obtain a global feature vector of the current video frame according to the current extracted features corresponding to the current video frame through the target discrimination network; perform similarity calculation based on the global feature vector and the initial feature vector, and determine the feature update method based on the vector similarity and the similarity threshold; update the memory features according to the feature update method and the current prediction mask to obtain updated memory features; perform target object segmentation on each video frame of the target video according to the updated memory features until the object segmentation of each video frame is completed.

[0064] In recent years, segmentation methods based on memory networks and feature matching have become one of the research hotspots in the segmentation of video foreground object sequences. Based on multi-level memory mechanisms and complex matching patterns, some researchers have proposed memory matching models, which have received widespread attention in the field of video foreground object sequence segmentation. STM (Spacetime Memory) uses a non-local matching algorithm to complete the matching between the features of the current video frame and the features of the target object in the memory network, so as to enhance the relevant features of the target object in the current video frame. STCN (Space-Time Contextual Network) further improves the matching algorithm and reuses the features encoded by the segmentation network into the memory network to reduce redundant encoding. SWEM (Space-Time Weighted Memory) adopts a multi-stage memory mode and uses memory features of different stages to enhance the segmentation effect of the foreground object sequence of the current video frame. AOT (Adaptive Object Tracking) uses a target re-identification algorithm to complete the synchronous segmentation of multiple foreground objects in the video. DeAOT (Decoupled Adaptive Object Tracking) further decouples feature matching and multi-target distinction, and uses a gated attention matching algorithm to enhance the segmentation effect of video foreground objects. ISVOS (Instance-Specific Video Object Segmentation) uses instance segmentation algorithms to enhance the segmentation of foreground objects. Cutie (Cut and Track Instance-level Estimation) uses a target-level matching algorithm to further reduce the impact of similar distractors in the same video frame on the segmentation of foreground objects.

[0065] Although these segmentation methods based on memory networks and feature matching have achieved good segmentation results, they all use fixed intervals to update the memory network, which makes it difficult to fully adjust the memory in a timely manner based on the changing characteristics of the object in the video, and it is easy to lose the key features of the object that appear in the video change process, which is crucial for the effective segmentation of targets that have undergone serious changes. At the same time, since it is impossible to accurately judge the change of the object, the memory network cannot timely integrate the redundant features of the foreground object when the change is small, and cannot make targeted enhancements to the key features that appear when the foreground object changes greatly, making it difficult to complete the effective segmentation of target objects in complex scene videos.

[0066] The present application provides a solution to identify the changing state of foreground objects within a unified segmentation framework, capture the key features of foreground objects during the change process, and dynamically adjust the memory features stored in the memory network, thereby improving the accuracy and robustness of video foreground object sequence segmentation in various scenarios.

[0067] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a foreground object segmentation device based on contrast learning that can realize the above functions. The following takes the foreground object segmentation device based on contrast learning as an execution subject as an example to illustrate this embodiment and the following embodiments.

[0068] Based on this, the embodiment of the present application provides a foreground object segmentation method based on contrast learning, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the foreground object segmentation method based on contrastive learning in this application.

[0069] In this embodiment, the foreground object segmentation method based on contrastive learning includes steps S10 to S50:

[0070] Step S10, generating a current prediction mask of the current video frame according to the current video frame features corresponding to the current video frame and the memory features stored in the memory network through the target segmentation network;

[0071] It should be noted that the method framework of this embodiment is composed of a target segmentation network, a target discrimination network and a memory network. The target segmentation network enhances the relevant features of the target object in the current video frame by matching the stored features in the target memory network to achieve the segmentation of the target object in the current input video frame; the target discrimination network determines the degree of change of the target object by calculating the feature similarity between different video frames to determine the memory update method in the memory network; the target memory network re-encodes the segmentation result and completes the dynamic update of short-term memory or long-term memory based on the discrimination signal generated by the discrimination network.

[0072] It can be understood that for the segmentation network, the training frame is first input into the encoder of the segmentation network to obtain the encoded current frame features, and the stored features in the memory network are matched with them to enhance the features related to the target object, and the enhanced current frame features are input into the decoder of the segmentation network, and the segmentation network is trained using mask true value supervision; then, the test frame is input into the segmentation network to generate a mask to complete the segmentation of the target object. In this embodiment, the trained segmentation network is the target segmentation network. The test frame involved in this embodiment is also the video frame involved in subsequent practical applications.

[0073] In a specific implementation, for the discriminant network, first, the training frame is input into the encoder of the discriminant network, and the encoded features are enhanced by the two-stage attention mechanism of target activation and semantic activation, and then the features are projected and multi-level contrastive learning is used to complete the training process of the discriminant network; then, the test frame is gradually input into the discriminant network to generate a feature vector, and the similarity between the feature vectors of different frames is calculated to determine the update method of the current frame in the memory network. In this embodiment, the trained discriminant network is the target discriminant network.

[0074] It should be noted that for the memory network, first, the segmentation result of the segmentation network for the training frame is input into the encoder of the memory network. After the encoding features of the segmentation network for the frame are fused, the features of the enhanced target object are obtained. According to the span of the video frame in the video, the current features are input into the fusion module of the short-term memory or the enhancement module of the long-term memory to complete the dynamic update of the memory network storage features; then, the segmentation result of the segmentation network for the test frame is input into the memory network, and according to the discrimination signal generated by the trained discriminant network for the changes of the target object in the current frame, the short-term memory or long-term memory is selected to be dynamically updated to prepare for the segmentation network to match and enhance the segmentation of subsequent frames.

[0075] It can be understood that the target video is a video that needs to be segmented, and the target video contains multiple video frames, and the current video frame is I t , t∈{1,2,...,T}, T represents the number of video frames corresponding to the target video. t Input to the target segmentation network, the target segmentation network is based on the current video frame I t The current video frame features and the memory features stored in the memory network to generate the current video frame I t The corresponding current prediction mask Y t .

[0076] In a feasible implementation, step S10 may include steps A11 to A13:

[0077] Step A11, extracting features of the current video frame through a feature extraction network in the target segmentation network to obtain current video frame features corresponding to the current video frame;

[0078] It should be noted that the current video frame I t Input to the target segmentation network, the target segmentation network uses the feature extraction network to analyze the current video frame I t Perform feature extraction to obtain the current video frame features

[0079]

[0080] Step A12, performing a matching operation based on the current video frame features and the memory features stored in the memory network to obtain target enhancement features.

[0081] It should be noted that the current video frame features and the memory features stored in the memory network are matched to obtain the current frame features of the target object enhancement, which is the target enhancement feature.

[0082]

[0083] Step A13, inputting the target enhancement feature into the decoder in the target segmentation network to generate a current prediction mask for the current video frame.

[0084] It should be noted that the target enhancement feature Input to the decoder in the target segmentation network to generate the current prediction mask Y corresponding to the target object in the current video frame t .

[0085] In a feasible implementation, step A12 may include steps B121 to B124:

[0086] Step B121, compressing the number of feature channels of the current video frame feature to generate a target query.

[0087] Step B122, determining the feature key and target value based on the memory features stored in the memory network.

[0088] Step B123, performing a similarity operation based on the target query and the feature key to obtain a similarity matrix.

[0089] Step B124, generating features according to the similarity matrix and the target value to obtain target enhanced features.

[0090] It should be noted that the target segmentation network compresses the current video frame feature F through projection operation. t The amount of feature channel data to generate the target query Determine the stored feature key based on the memory features stored in the memory network and the stored target value Target query and feature keys Perform similarity calculation to obtain the similarity matrix. After normalizing the similarity matrix, calculate the target value Weighted to generate the current frame features of the target object enhancement Where j∈{1,...,N}, N represents the number of stored memory frames. The specific calculation process is: Among them, SoftMax(·) represents the SoftMax operator, and <·,·> represents the L2 similarity calculation process.

[0091] The above is only one feasible implementation of step S10 provided in this embodiment, and this embodiment does not specifically limit the specific implementation of step S10.

[0092] Step S20, obtaining a global feature vector of the current video frame according to the current extracted features corresponding to the current video frame through a target discrimination network.

[0093] It should be noted that the target discrimination network uses the feature extraction network to t Perform feature extraction to obtain the current extracted features The feature is enhanced and projected to obtain the current video frame I t The global eigenvector of In this embodiment, the global feature vector is used for subsequent similarity calculation between different frames and to generate a discrimination signal of the memory network update mode.

[0094] Step S30, performing similarity calculation based on the global feature vector and the initial feature vector, and determining a feature update method based on the vector similarity and a similarity threshold.

[0095] It should be noted that the initial feature vector refers to the initial feature vector of the i-th change segment The global eigenvector g t and the initial eigenvector g i A similarity operation is performed to obtain the vector similarity between the two.

[0096] It can be understood that when the vector similarity is compared with the similarity threshold, if the vector similarity is lower than the similarity threshold, it means that the object has changed dramatically and the i+1th change segment needs to be opened. At the same time, the long-term memory should be updated in the memory network. At this time, the feature update method is the long-term memory update method; otherwise, the i-th change segment is still maintained, and the short-term memory is updated in the memory network. At this time, the feature update method is the short-term memory update method.

[0097] In a feasible implementation, step S30 may include steps A31 to A34:

[0098] Step A31, randomly selecting video frames from multiple video clips corresponding to the target video to obtain multiple extracted frames;

[0099] Step A32, determining the global feature vector of each extracted frame through the target discrimination network;

[0100] Step A33, calculating similarities between adjacent video frames based on the global feature vectors of each extracted frame to obtain multiple target similarities;

[0101] Step A34, performing mean calculation based on the similarities of each target to obtain a similarity threshold.

[0102] It should be noted that before object segmentation is performed on the target video, the target video will be divided into K video segments, and a frame will be randomly selected from each video segment to obtain K video frames. The global feature vector g corresponding to the K video frames will be directly calculated using the target discrimination network, and then the target similarity between the global feature vectors of adjacent video frames will be calculated in turn, and the average of all target similarities will be used as the similarity threshold for subsequent discrimination.

[0103] Step S40, updating the memory feature according to the feature update mode and the current prediction mask to obtain an updated memory feature;

[0104] It should be noted that the dynamic update of the memory network storage process is mainly realized by the similarity fusion module and the change highlighting module. When the target discriminant network does not need to open a new change segment, the feature update method is the short-term memory update method. The similarity fusion module encodes the features corresponding to the current prediction mask: the memory network encoding feature M t Fusion is performed to eliminate redundant features in memory and update the short-term memory in the current memory features. Thus, the updated memory features are obtained; when the discriminant network generates a signal to start a new change segment, the feature update method is the long-term memory update method, and the change highlighting module compares the current encoding feature M t and the memory network encoding feature M at the beginning of the segment i , highlight the change features and use them as the long-term memory features of the current change segment Thus, the updated memory features are obtained.

[0105] Step S50 , performing target object segmentation on each video frame of the target video according to the updated memory features, until the object segmentation of each video frame is completed.

[0106] It should be noted that the updated memory features participate in the mask prediction process of the next video frame in the target segmentation network, and the target discrimination network determines the feature update method of the memory network based on the global feature vector. The memory network updates the currently stored memory features according to the feature update method, and repeats all the above steps until the target object is segmented for all video frames in the target video. In this embodiment, the memory features corresponding to the first video frame are obtained through annotation, and the subsequent memory features are updated and stored based on the previous video frame in combination with the corresponding update method.

[0107] In this embodiment, the target segmentation network generates the current prediction mask of the current video frame according to the current video frame features corresponding to the current video frame and the memory features stored in the memory network; the target discrimination network obtains the global feature vector of the current video frame according to the current extracted features corresponding to the current video frame; similarity is calculated based on the global feature vector and the initial feature vector, and the feature update method is determined based on the vector similarity and the similarity threshold; the memory features are updated according to the feature update method and the current prediction mask to obtain the updated memory features; the target object is segmented for each video frame of the target video according to the updated memory features until the object segmentation of each video frame is completed. Through the above method, the change state of the foreground object is judged within a unified segmentation framework, the key features of the foreground object in the change process are captured, and the memory features stored in the memory network are dynamically adjusted, thereby improving the accuracy and robustness of the segmentation of the video foreground object sequence in various scenarios.

[0108] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 2 In the foreground object segmentation method based on contrastive learning, step S20 further includes steps S21 to S25:

[0109] Step S21, extracting features of the current video frame through the feature extraction network in the target discrimination network to obtain current extracted features corresponding to the current video frame.

[0110] It should be noted that the target discrimination network uses the feature extraction network to t Perform feature extraction to obtain the current extracted features

[0111] Step S22, mapping the currently extracted features to obtain a first query, a first key, and a first value.

[0112] It should be noted that the target discriminant network obtains the current extracted feature X t After that, it will further perform the two-stage attention mechanism operation of target activation and semantic activation to obtain the global feature vector g of the current video frame t .

[0113] It can be understood that when performing target activation, the target discrimination network first extracts the current feature X t Mapping gets the first query First Key

[0114] Step S23: obtaining a target activation feature according to the first query, the first key and the first value.

[0115] It should be noted that the target discriminant network performs a similarity operation on the first query and the first key, normalizes the similarity matrix, and then weights the first value to obtain the target activated feature, thereby obtaining the target activation feature The specific calculation process is:

[0116]

[0117] Step S23, performing semantic activation according to the second query, the second key and the second value corresponding to the target activation feature to obtain a semantic activation feature.

[0118] It should be noted that the target activation feature X obtained after target activation t,s After further mapping, we get the second query Second key and the second value The similarity operation is performed on the second query and the second key, and the similarity matrix is ​​normalized and the second value is weighted to obtain the target activated feature, thereby obtaining the semantic activation feature. The specific calculation process is:

[0119] Step S24, obtaining a global feature vector of the current video frame according to the semantic activation feature.

[0120] It should be noted that the semantic activation feature X t,sc After a layer of global average pooling and a multivariate perceptron, the global feature vector g of the current video frame is obtained. t .

[0121] It can be understood that in order to make the global feature vector g generated by the target discriminant network t It can better represent the meaning of the target object. During the training process, the contrast learning loss function at the frame level, object level and pixel level is used to supervise the features. The specific process is as follows: Frame-level contrast learning loss function: Use the encoder parameters of the trained segmentation network to initialize the encoder parameters of the discriminant network. During training, the input video is divided into K segments, and one frame is randomly sampled from each segment, totaling K frames; then one frame is randomly sampled from another K videos, totaling K frames; 2K video frames are regarded as a training batch. For the K frames randomly sampled from the same video, first use the encoder of the discriminant network to encode them to obtain features. In order to further balance the semantic contribution of the target object and the background, the number of target object pixels in each frame is calculated using the true value mask. O and the number of background pixels nB , and then a layer of global weighted pooling is performed to obtain K global vectors Then each global vector is considered as a targeting quantity Calculate the similarity between the vectors and the other K-1 vectors, and regard the one with the highest similarity as the positive sample pair of the target quantity, and the one with the lowest similarity as the negative sample pair of the target quantity. The overall operation process can be expressed by the following formula: Among them, O and B represent the target object and background area respectively. Each target video frame I α Respectively and I β , I γ Constitute a positive and negative sample pair. Then I γ Together with K frames randomly sampled from another K videos, they form a negative sample library N. After clarifying the positive and negative samples corresponding to each video frame, the target activation and semantic activation pair encoding are used to obtain the feature Enhance and finally map the features into a global feature vector g t , and the frame-level contrastive learning supervision is: Among them, τ is used as a hyperparameter to control the training convergence rate.

[0122] In this embodiment, the target level contrast learning loss function is as follows: after clarifying the relationship between positive and negative samples between video frames, the target level contrast learning loss function is constructed to further improve the semantic attention of the global feature vector to the target object. The specific process is as follows: α And its corresponding positive sample I β And the region O where the target object is located in the negative sample library N performs average pooling of the local region of its encoding feature X in turn, and obtains the target level vectors o, o respectively. + and - ∈N o , and perform target-level contrastive learning supervision on them as follows: Pixel-level contrast learning loss function: Since video object segmentation is a pixel-intensive task, it is necessary to further specify the supervised learning at the pixel level. α A pixel-level feature vector p in the positive and negative sample pairs I β and I γ Find the pixel-level feature vector that is most similar to it. The calculation process is as follows: where p s That is, the positive sample p of the feature vector p + , and I β and I γ The remaining pixels p - Together they form the negative sample library N p , and conduct pixel-level contrast learning supervision on them as follows:

[0123] This embodiment extracts features from the current video frame through the feature extraction network in the target discrimination network to obtain the current extracted features corresponding to the current video frame; maps the current extracted features to obtain a first query, a first key, and a first value; obtains a target activation feature according to the first query, the first key, and the first value; performs semantic activation according to the second query, the second key, and the second value corresponding to the target activation feature to obtain a semantic activation feature; obtains the global feature vector of the current video frame according to the semantic activation feature. In the above manner, the global feature vector of the current video frame can be accurately obtained, thereby laying a foundation for the subsequent update of the memory feature.

[0124] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 3 , step S40 in the foreground object segmentation method based on contrastive learning further includes steps S41 to S45:

[0125] Step S41, when the feature updating mode is the short-term memory updating mode, the initial coding feature of the segment is obtained.

[0126] It should be noted that the dynamic update of the memory network storage process is mainly realized by the similarity fusion module and the change highlighting module. When the discriminant network does not need to open a new change segment, the similarity fusion module encodes the memory network feature M t Fusion is performed to eliminate redundant features in memory and update the current short-term memory When the discriminant network generates a signal to start a new change segment, the change highlighting module compares the current encoding feature M t and the memory network encoding feature M at the beginning of the segment i , highlight the change features and use them as the long-term memory features of the current change segment

[0127] It can be understood that when the feature update mode is the short-term memory update mode, it means that the short-term memory in the memory feature stored in the memory network needs to be updated. At this time, the initial encoding feature M of the change segment is obtained. i , which is the initial encoding feature of the fragment.

[0128] Step S42, determining the current long-term memory according to the memory feature, and obtaining the current encoding feature according to the current prediction mask and the current video frame feature.

[0129] It should be noted that the current long-term memory is determined based on the memory features stored in the memory network, and the memory network uses the feature extraction network to extract the current prediction mask Y t Re-encode and fuse the current video frame features F t , get the memory encoding features of the current video frame In this embodiment, the current encoding feature refers to the memory encoding feature M of the current video frame. t .

[0130] Step S43, performing feature splicing according to the current coding feature and the initial coding feature of the segment to obtain a splicing feature;

[0131] Step S44, performing multi-scale fusion on the splicing features to obtain target short-term memory;

[0132] Step S45, obtaining updated memory features according to the target short-term memory and the current long-term memory.

[0133] It should be noted that the similarity fusion module in the memory network converts the current encoding feature M t and the fragment initial encoding feature M i Splicing (Cat) to obtain splicing features. Then multi-scale fusion is performed on the splicing features. Multi-scale fusion contains 4 branches, each of which contains an average pooling layer (AP) and a deformation convolution layer (DC), so that the features can pay more attention to the deformation information of the object. The overall fusion process can be expressed as follows: j =DC j (AP j (Cat(M t ,M i ))),j∈{1,2,3,4}, then concatenate the outputs of the four branches and fuse them with a 3×3 depthwise separable convolution to generate the final short-term memory. Based on current long-term memory and target short-term memory Together they constitute the updated memory features.

[0134] In a feasible implementation manner, step S40 may further include steps A41 to A45:

[0135] Step A41, when the feature update mode is a long-term memory update mode, determining the current short-term memory according to the memory feature, and obtaining the current encoding feature according to the current prediction mask and the current video frame feature;

[0136] When the feature update mode is the long-term memory update mode, it means that the long-term memory in the memory features stored in the memory network needs to be updated. At this time, the current short-term memory is determined according to the memory features stored in the memory network. At the same time, the memory network uses the feature extraction network to update the current prediction mask Y t Re-encode and fuse the current video frame features F t , get the memory encoding feature M of the current video frame t .

[0137] Step A42, performing similarity calculation based on the current encoding feature and the initial encoding feature of the segment to obtain a similarity matrix;

[0138] Step A43, performing feature processing according to the similarity matrix and the initial encoding features of the fragments to determine the adjustment weight;

[0139] Step A44, performing feature processing on the current encoding feature according to the adjustment weight to obtain a target long-term memory;

[0140] Step A45, obtaining updated memory characteristics according to the target long-term memory and the current short-term memory.

[0141] It should be noted that in order to further highlight and amplify the feature changes of the target object at the beginning and end of a change segment, the change highlighting module in the memory network first encodes the current encoding feature M t and the fragment initial encoding feature M i Perform similarity calculation to obtain the similarity matrix. After normalizing the similarity matrix, the initial encoding feature M of the segment is i The specific process can be expressed as follows: in Indicates the current encoding feature M t Then concatenate the features and feature differences and use a 1×1 convolution layer (Conv) to reduce the number of channels of the concatenated features, and normalize the features. The result is the adjusted weight. Adjust the current encoding feature M by adjusting the weight t , in order to enhance the change characteristics of the target object, the specific process can be expressed as follows: in represents element-wise product, The long-term memory corresponding to the current change segment, that is, the target long-term memory Based on current short-term memory and target long-term memory Together they constitute the updated memory features.

[0142] In this embodiment, when the feature update mode is the long-term memory update mode, the current short-term memory is determined according to the memory feature, and the current coding feature is obtained according to the current prediction mask and the current video frame feature; similarity is calculated according to the current coding feature and the initial coding feature of the fragment to obtain a similarity matrix; feature processing is performed according to the similarity matrix and the initial coding feature of the fragment to determine the adjustment weight; feature processing is performed on the current coding feature according to the adjustment weight to obtain the target long-term memory; and the updated memory feature is obtained according to the target long-term memory and the current short-term memory. In the above manner, a new long-term memory is generated by highlighting the difference, thereby completing the dynamic update of the memory network storage memory, which can effectively improve the effect of target object segmentation in subsequent frames.

[0143] This embodiment obtains the initial coding features of the fragment when the feature update mode is the short-term memory update mode; determines the current long-term memory according to the memory features, and obtains the current coding features according to the current prediction mask and the current video frame features; performs feature splicing according to the current coding features and the initial coding features of the fragment to obtain splicing features; performs multi-scale fusion on the splicing features to obtain the target short-term memory; and obtains the updated memory features according to the target short-term memory and the current long-term memory. Through the above method, through the above method, a new short-term memory is generated through similar fusion, thereby completing the dynamic update of the memory network storage memory, which can effectively improve the effect of target object segmentation in subsequent frames.

[0144] For example, in order to help understand the implementation process of the foreground object segmentation method based on contrast learning obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 4 , Figure 4 A brief flowchart of a foreground object segmentation method based on contrastive learning is provided. Specifically:

[0145] The foreground object segmentation method based on contrastive learning of this embodiment is composed of a segmentation network, a discriminant network and a memory network. S1. The segmentation network enhances the relevant features of the target object in the current video frame by matching the stored features in the memory network to achieve the segmentation of the target object in the current input video frame; the discriminant network determines the degree of change of the target object by calculating the feature similarity between different video frames to determine the memory update method in the memory network; the memory network re-encodes the segmentation result and completes the dynamic update of short-term memory or long-term memory based on the discriminant signal generated by the discriminant network.

[0146] S2. For the segmentation network, first, the training frame is input into the encoder of the segmentation network to obtain the encoded current frame features, and the stored features in the memory network are matched with them to enhance the features related to the target object, and the enhanced current frame features are input into the decoder of the segmentation network, and the segmentation network is trained using mask true value supervision; then, the test frame is input into the segmentation network to generate a mask to complete the segmentation of the target object.

[0147] S3. For the discriminant network, first, the training frame is input into the encoder of the discriminant network, and the encoded features are enhanced by the two-stage attention mechanism of target activation and semantic activation. Then, the features are projected and multi-level contrastive learning is used to complete the training process of the discriminant network; then, the test frames are gradually input into the discriminant network to generate feature vectors, and the similarity between the feature vectors of different frames is calculated to determine the update method of the current frame in the memory network.

[0148] S4. For the memory network, first, the segmentation result of the training frame by the segmentation network is input into the encoder of the memory network. After fusing the encoding features of the frame by the segmentation network, the features of the enhanced target object are obtained. According to the span of the video frame in the video, the current features are input into the fusion module of the short-term memory or the enhancement module of the long-term memory to complete the dynamic update of the memory network storage features. Then, the segmentation result of the test frame by the segmentation network is input into the memory network. According to the discrimination signal of the target object change in the current frame generated by the trained discrimination network, the short-term memory or long-term memory is selected to be dynamically updated to prepare for the segmentation network to match and enhance the segmentation of subsequent frames.

[0149] This embodiment uses a framework of contrastive learning and adaptive memory update to keenly perceive the degree of change of the target object, accurately capture the key features of the target object in the change process, and dynamically adjust the storage features in the memory network, thereby improving the accuracy of target object segmentation in complex scenes and having a more robust segmentation performance when dealing with various complex scenes.

[0150] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the foreground object segmentation method based on contrast learning of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.

[0151] This application also provides a foreground object segmentation device based on contrast learning, please refer to Figure 5 , the foreground object segmentation device based on contrastive learning comprises:

[0152] A generation module 10, configured to generate a current prediction mask of the current video frame according to a current video frame feature corresponding to the current video frame and a memory feature stored in the memory network through a target segmentation network;

[0153] A processing module 20, configured to obtain a global feature vector of the current video frame according to a current extracted feature corresponding to the current video frame through a target discrimination network;

[0154] A calculation module 30, used for performing similarity calculation based on the global feature vector and the initial feature vector, and determining a feature update method based on the vector similarity and a similarity threshold;

[0155] An updating module 40, configured to update the memory feature according to the feature updating mode and the current prediction mask to obtain an updated memory feature;

[0156] The segmentation module 50 is used to segment the target object in each video frame of the target video according to the updated memory features until the object segmentation of each video frame is completed.

[0157] Optionally, the generating module 10 is further used for:

[0158] Extract features of the current video frame through a feature extraction network in the target segmentation network to obtain current video frame features corresponding to the current video frame;

[0159] Performing a matching operation based on the current video frame feature and the memory feature stored in the memory network to obtain a target enhancement feature;

[0160] The target enhancement feature is input into a decoder in the target segmentation network to generate a current prediction mask for a current video frame.

[0161] Optionally, the generating module 10 is further used for:

[0162] Compressing the number of feature channels of the current video frame feature to generate a target query;

[0163] Determine feature keys and target values ​​according to memory features stored in the memory network;

[0164] Perform similarity calculation based on the target query and the feature key to obtain a similarity matrix;

[0165] Features are generated according to the similarity matrix and the target value to obtain target enhanced features.

[0166] Optionally, the processing module 20 is further used for:

[0167] Extract features of the current video frame through a feature extraction network in the target discrimination network to obtain current extracted features corresponding to the current video frame;

[0168] Mapping the currently extracted feature to obtain a first query, a first key, and a first value;

[0169] Obtain a target activation feature according to the first query, the first key, and the first value;

[0170] Performing semantic activation according to a second query, a second key, and a second value corresponding to the target activation feature to obtain a semantic activation feature;

[0171] A global feature vector of the current video frame is obtained according to the semantic activation feature.

[0172] Optionally, the calculation module 30 is further used for:

[0173] Randomly selecting video frames from multiple video clips corresponding to the target video to obtain multiple extracted frames;

[0174] Determining the global feature vector of each extracted frame through the target discrimination network;

[0175] The similarity between adjacent video frames is calculated based on the global feature vector of each extracted frame to obtain multiple target similarities;

[0176] The similarity threshold is obtained by calculating the mean value of each target similarity.

[0177] Optionally, the updating module 40 is further configured to:

[0178] When the feature updating mode is a short-term memory updating mode, obtaining the initial encoding feature of the fragment;

[0179] Determining a current long-term memory according to the memory feature, and obtaining a current encoding feature according to the current prediction mask and the current video frame feature;

[0180] Perform feature splicing according to the current coding feature and the initial coding feature of the fragment to obtain a splicing feature;

[0181] Performing multi-scale fusion on the splicing features to obtain target short-term memory;

[0182] An updated memory feature is obtained according to the target short-term memory and the current long-term memory.

[0183] Optionally, the updating module 40 is further configured to:

[0184] When the feature update mode is a long-term memory update mode, determining the current short-term memory according to the memory feature, and obtaining the current encoding feature according to the current prediction mask and the current video frame feature;

[0185] Calculate similarity based on the current encoding feature and the initial encoding feature of the segment to obtain a similarity matrix;

[0186] Perform feature processing according to the similarity matrix and the initial encoding features of the fragment to determine an adjustment weight;

[0187] Performing feature processing on the current encoding feature according to the adjustment weight to obtain a target long-term memory;

[0188] An updated memory feature is obtained according to the target long-term memory and the current short-term memory.

[0189] The foreground object segmentation device based on contrastive learning provided by the present application adopts the foreground object segmentation method based on contrastive learning in the above-mentioned embodiment, and can solve the technical problem of how to improve the accuracy and robustness of the segmentation of the video foreground object sequence. Compared with the prior art, the beneficial effects of the foreground object segmentation device based on contrastive learning provided by the present application are the same as the beneficial effects of the foreground object segmentation method based on contrastive learning provided by the above-mentioned embodiment, and the other technical features of the foreground object segmentation device based on contrastive learning are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0190] The present application provides a foreground object segmentation device based on contrastive learning, and the foreground object segmentation device based on contrastive learning includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the foreground object segmentation method based on contrastive learning in the above-mentioned embodiment 1.

[0191] Reference below Figure 6 , which shows a schematic diagram of the structure of a foreground object segmentation device based on contrastive learning suitable for implementing the embodiment of the present application. The foreground object segmentation device based on contrastive learning in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The foreground object segmentation device based on contrastive learning shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0192] like Figure 6As shown, the foreground object segmentation device based on contrastive learning may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the foreground object segmentation device based on contrastive learning are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems may be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the foreground object segmentation device based on contrastive learning to communicate with other devices wirelessly or by wire to exchange data. Although the foreground object segmentation device based on contrastive learning with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0193] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0194] The foreground object segmentation device based on contrastive learning provided by the present application adopts the foreground object segmentation method based on contrastive learning in the above embodiment, which can solve the technical problem of how to improve the accuracy and robustness of the segmentation of the video foreground object sequence. Compared with the prior art, the beneficial effects of the foreground object segmentation device based on contrastive learning provided by the present application are the same as the beneficial effects of the foreground object segmentation method based on contrastive learning provided by the above embodiment, and the other technical features of the foreground object segmentation device based on contrastive learning are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0195] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0196] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0197] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the foreground object segmentation method based on contrastive learning in the above-mentioned embodiment.

[0198] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0199] The computer-readable storage medium may be included in the foreground object segmentation device based on contrastive learning; or may exist independently without being assembled into the foreground object segmentation device based on contrastive learning.

[0200] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a foreground object segmentation device based on contrastive learning, the foreground object segmentation device based on contrastive learning: generates a current prediction mask of the current video frame according to the current video frame features corresponding to the current video frame and the memory features stored in the memory network through a target segmentation network; obtains a global feature vector of the current video frame according to the current extracted features corresponding to the current video frame through a target discrimination network; performs similarity calculation based on the global feature vector and the initial feature vector, and determines a feature update method based on the vector similarity and a similarity threshold; updates the memory features according to the feature update method and the current prediction mask to obtain updated memory features; and performs target object segmentation on each video frame of the target video according to the updated memory features until the object segmentation of each video frame is completed.

[0201] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0202] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0203] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0204] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the foreground object segmentation method based on contrastive learning, and can solve the technical problem of how to improve the accuracy and robustness of video foreground object sequence segmentation. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the foreground object segmentation method based on contrastive learning provided by the above embodiment, and will not be repeated here.

[0205] The present application also provides a computer program product, including a computer program, which implements the steps of the foreground object segmentation method based on contrast learning when executed by a processor.

[0206] The computer program product provided by the present application can solve the technical problem of how to improve the accuracy and robustness of the segmentation of the foreground object sequence of the video. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the foreground object segmentation method based on contrastive learning provided by the above embodiment, which will not be described in detail here.

[0207] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A foreground object segmentation method based on contrastive learning, characterized in that: The foreground object segmentation method based on contrastive learning includes: Generate a current prediction mask of the current video frame according to the current video frame features corresponding to the current video frame and the memory features stored in the memory network through the target segmentation network; Obtaining a global feature vector of the current video frame according to the current extracted features corresponding to the current video frame through a target discrimination network; Calculating similarity based on the global feature vector and the initial feature vector, and determining a feature update method based on the vector similarity and a similarity threshold; updating the memory feature according to the feature update mode and the current prediction mask to obtain an updated memory feature; Target object segmentation is performed on each video frame of the target video according to the updated memory features until the object segmentation of each video frame is completed.

2. The method according to claim 1, characterized in that The step of generating a current prediction mask of the current video frame according to the current video frame features corresponding to the current video frame and the memory features stored in the memory network through the target segmentation network comprises: Extract features of the current video frame through a feature extraction network in the target segmentation network to obtain current video frame features corresponding to the current video frame; Performing a matching operation based on the current video frame feature and the memory feature stored in the memory network to obtain a target enhancement feature; The target enhancement feature is input into a decoder in the target segmentation network to generate a current prediction mask for a current video frame.

3. The method according to claim 2, characterized in that The step of performing a matching operation based on the current video frame feature and the memory feature stored in the memory network to obtain the target enhanced feature comprises: Compressing the number of feature channels of the current video frame feature to generate a target query; Determine feature keys and target values ​​according to memory features stored in the memory network; Perform similarity calculation based on the target query and the feature key to obtain a similarity matrix; Features are generated according to the similarity matrix and the target value to obtain target enhanced features.

4. The method according to claim 1, characterized in that The step of obtaining the global feature vector of the current video frame according to the current extracted features corresponding to the current video frame through the target discrimination network includes: Extract features of the current video frame through a feature extraction network in the target discrimination network to obtain current extracted features corresponding to the current video frame; Mapping the currently extracted feature to obtain a first query, a first key, and a first value; Obtain a target activation feature according to the first query, the first key, and the first value; Performing semantic activation according to a second query, a second key, and a second value corresponding to the target activation feature to obtain a semantic activation feature; A global feature vector of the current video frame is obtained according to the semantic activation feature.

5. The method according to claim 1, characterized in that Before the step of calculating similarity based on the global feature vector and the initial feature vector and determining the feature update method based on the vector similarity and the similarity threshold, the method further includes: Randomly selecting video frames from multiple video clips corresponding to the target video to obtain multiple extracted frames; Determining the global feature vector of each extracted frame through the target discrimination network; The similarity between adjacent video frames is calculated based on the global feature vector of each extracted frame to obtain multiple target similarities; The similarity threshold is obtained by calculating the mean value of each target similarity.

6. The method according to claim 1, characterized in that The step of updating the memory feature according to the feature update mode and the current prediction mask to obtain the updated memory feature comprises: When the feature updating mode is a short-term memory updating mode, obtaining the initial encoding feature of the fragment; Determining a current long-term memory according to the memory feature, and obtaining a current encoding feature according to the current prediction mask and the current video frame feature; Perform feature splicing according to the current coding feature and the initial coding feature of the fragment to obtain a splicing feature; Performing multi-scale fusion on the splicing features to obtain target short-term memory; An updated memory feature is obtained according to the target short-term memory and the current long-term memory.

7. The method according to claim 1, characterized in that The step of updating the memory feature according to the feature update mode and the current prediction mask to obtain the updated memory feature comprises: When the feature update mode is a long-term memory update mode, determining the current short-term memory according to the memory feature, and obtaining the current encoding feature according to the current prediction mask and the current video frame feature; Calculate similarity based on the current encoding feature and the initial encoding feature of the segment to obtain a similarity matrix; Perform feature processing according to the similarity matrix and the initial encoding features of the fragment to determine an adjustment weight; Performing feature processing on the current encoding feature according to the adjustment weight to obtain a target long-term memory; An updated memory feature is obtained according to the target long-term memory and the current short-term memory.

8. A foreground object segmentation device based on contrastive learning, characterized in that: The foreground object segmentation device based on contrastive learning comprises: A generation module, configured to generate a current prediction mask of the current video frame according to a current video frame feature corresponding to the current video frame and a memory feature stored in the memory network through a target segmentation network; A processing module, configured to obtain a global feature vector of the current video frame according to a current extracted feature corresponding to the current video frame through a target discrimination network; A calculation module, used for performing similarity calculation based on the global feature vector and the initial feature vector, and determining a feature update method based on the vector similarity and a similarity threshold; An updating module, used for updating the memory feature according to the feature updating mode and the current prediction mask to obtain an updated memory feature; The segmentation module is used to segment the target object in each video frame of the target video according to the updated memory features until the object segmentation of each video frame is completed.

9. A foreground object segmentation device based on contrastive learning, characterized in that: The foreground object segmentation device based on contrastive learning includes: a memory, a processor, and a foreground object segmentation program based on contrastive learning stored in the memory and executable on the processor, wherein the foreground object segmentation program based on contrastive learning is configured to implement the foreground object segmentation method based on contrastive learning as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a foreground object segmentation program based on contrastive learning, and when the foreground object segmentation program based on contrastive learning is executed by the processor, the foreground object segmentation method based on contrastive learning as described in any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Video scene segmentation method, device and equipment and computer readable storage medium

    CN114283351A

  • Data processing method and device, equipment, storage medium and computer program product

    CN115115981A

  • Semi-supervised target segmentation method based on global matching and historical frame feature optimization

    CN117975001A