A fast real-time video object segmentation method based on bimodal interaction and state feedback

Abstract

The present invention discloses a fast real-time video target segmentation method based on bimodal interaction and state feedback, comprising the following steps: step 1, model training, using a server to train a network model, optimizing network parameters by reducing the network loss function until the network converges, and obtaining the network weights of the fast real-time video target segmentation method based on bimodal interaction and state feedback; step 2, model inference, using the network weights obtained in the training phase to segment the target given in the first frame in a new video sequence. Through bimodal feature interaction, the learning of target appearance representation is strengthened, and the interference of targets with similar appearance features and similar colors in the background is weakened, so that the system can accurately segment targets in complex environments; and through state feedback, the effective capacity of the memory is fixed and the features in the memory are adaptively updated, so that the system can effectively use the target information in the historical frame, and has a high inference speed and memory utilization efficiency.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, and in particular relates to a fast real-time target segmentation method based on dual-modal interaction and state feedback. Background Art

[0002] Video target segmentation is one of the basic tasks in the field of computer vision and has important application value and significance in real life, such as video editing and autonomous driving. Driven by deep learning, video target segmentation has made great progress. This technology is mainly aimed at semi-supervised video target segmentation tasks. During testing, the user is required to provide the target mask of the first frame (usually) or key frame, and the algorithm automatically segments the remaining video frames. Due to problems such as target occlusion, deformation, motion blur and scale transformation, the most challenging aspect of the semi-supervised video target segmentation task is how to learn a stable target appearance.

[0003] At present, the mainstream algorithms for semi-supervised video object segmentation tasks can be roughly divided into two categories, online learning methods and offline learning methods. Online learning methods fine-tune the model according to the historical frame (usually the first frame) and its mask during the inference process, so that the model learns the appearance of the target in the current segmented video sequence, which can effectively improve the quality of the segmentation results, but the speed is usually slow; offline learning methods do not update model parameters during the inference process, usually combined with target tracking and other technologies, or use various matching methods to learn the target appearance offline, which is usually faster than online learning methods. Recently, matching-based methods have made great progress in the semi-supervised video object segmentation task. These methods use memory to store historical frame features, which greatly improves the quality of segmentation results. However, their real-time performance is not enough to meet practical use; and how to effectively organize the features in the memory to avoid problems such as memory overflow is still under exploration. In addition, the current mainstream methods do not make full use of the rich edge texture information contained in the target mask, which can help strengthen the learning of the target appearance and reduce the interference of background objects, thus causing a lot of information waste. Summary of the invention

[0004] To address the above problems, we propose a fast real-time video target segmentation method based on bimodal interaction and state feedback. The purpose of bimodal interaction is to enhance the learning of target appearance and reduce background interference through the mutual blending of image features and mask features to improve the segmentation quality; the purpose of state feedback is to provide the system with the state score of the segmentation result and assist in organizing the features in the memory to improve the spatiotemporal efficiency of the entire system.

[0005] The present invention aims at the problems in the existing semi-supervised video target segmentation method based on deep learning, such as the rough contour of the segmentation result caused by insufficient use of the mask, the possible memory overflow caused by the gradual increase of features stored in the memory, and the slow reasoning speed. A fast real-time target segmentation method based on bimodal interaction and state feedback is proposed. The purpose is to strengthen the learning of the target appearance representation through bimodal feature interaction, reduce the interference of targets with similar appearance features and similar colors in the background, so that the system can accurately segment the target in a complex environment; and fix the effective capacity of the memory through state feedback and adaptively update the features in the memory, so that the system can effectively use the target information in the historical frame, and have a high inference speed and memory utilization efficiency.

[0006] In order to achieve the above-mentioned invention object, the present invention adopts the following technical scheme: A fast real-time video target segmentation method based on dual-modal interaction and state feedback, comprising the following steps:

[0007] Step 1: Model training: Use the server to train the network model, optimize the network parameters by reducing the network loss function until the network converges, and obtain the network weights for fast real-time video target segmentation based on bimodal interaction and state feedback;

[0008] Step 2: Model inference: Use the network weights obtained in the training phase to segment the target given in the first frame in the new video sequence.

[0009] Furthermore, the model training in step 1 specifically includes the following steps:

[0010] S11, using the server to execute a training video segment generating unit to generate a training video segment of length T, where T ≥ 2;

[0011] S12, using the server to execute the query encoding unit to extract the query image key-value encoding pair, the current frame image is I t , the query image key-value encoding pair is t represents the time, t>1, Q represents the query image;

[0012] S13, using the server to execute the reference encoding unit, using the dual-branch residual network with unshared parameters of the protection image branch and the mask branch, extracting the reference frame image features and the reference frame target mask features and performing dual-modal interaction, the previous frame image of the reference frame image is I t-1 , the target mask prediction result of the previous frame of the reference frame target mask is M t-1 , output reference key-value coding pairs and stored in a memory, M refers to the maximum value of the equivalent capacity of the memory, and R represents a reference image;

[0013] S14, using the server to execute a mask reconstruction unit to reconstruct the target mask prediction result of the mask branch input into step S13, and output the reconstructed target mask;

[0014] S15, using the server to execute the state estimation unit, perform state estimation on the target mask prediction result input in step S13, provide state feedback, and output the predicted state score s t-1 ;

[0015] S16, using the server to execute the matching unit, according to the query image key-value encoding To retrieve the information in the historical frame features in the memory and obtain the final matching features;

[0016] S17, using the server to execute the decoding unit and output the final segmentation result M of the query frame t ;

[0017] S18, using the server to perform network training in an end-to-end manner; specifically, the segmentation loss function L s , reconstruction loss function L r And the state evaluation loss function L e Combine and use adaptive weights to automatically balance the loss functions of each part to obtain the final total loss function L total ; The mathematical expression of each loss is:

[0018] L s (Y t ,M t )=L ce (Y t ,M t )+α·L IoU (Y t ,M t )

[0019] L r (Y t-1 ,M r,t-1 )=L ce (Y t-1 ,M r,t-1 )+α·L IoU (Y t-1 ,M r,t-1 )

[0020]

[0021]

[0022] in, represents the cross entropy loss;

[0023] Y represents the Mask IoU loss; t-1 ,Y t Respectively represent the target mask truth value of the previous frame and the target mask truth value of the current frame; M t-1 ,M t Respectively represent the target mask prediction result of the previous frame and the target mask prediction result of the current frame; M r,t-1 Represents the reconstructed value of the target mask prediction result of the previous frame; s t-1 Indicates M t-1 The state prediction score of ; Ω represents the set of all pixels in the target mask; T represents the length of the training video segment; α is a hyperparameter; w1, w2 are learnable parameters;

[0024] S19, using the server to optimize the objective function, the objective function adopts the total loss function L in step S18 total , obtain the local optimal network parameters as the network weights for fast real-time video object segmentation based on bimodal fusion and state feedback.

[0025] Furthermore, the value of T in step S11 is 3, and the value of α in step S18 is 1.

[0026] Furthermore, the step S11 is specifically as follows: randomly extracting T images at intervals from any video of the plurality of video data sets, and performing T different affine transformations on the T images respectively, where the affine transformations include translation, scaling, flipping, rotation and shearing, to form training video clips;

[0027] Alternatively, an image is randomly selected from the image dataset and subjected to T different affine transformations to form a training video clip.

[0028] Furthermore, the step S13 is specifically as follows: using a dual-branch residual network including an image branch and a mask branch that do not share parameters to extract features from the input reference frame image and the reference frame target mask prediction result respectively; then adding the features of each stage of the mask branch and the features of the corresponding stage of the image branch through a squeeze excitation block (SEblock); then injecting the added features into the image branch; and finally the image branch outputs a reference key-value coding pair. Reference key-value coding pairs are stored directly into memory.

[0029] Furthermore, the step S16 is specifically: using the query image key-value encoding In The historical frame features in the memory Calculate the similarity and normalize it as a weight for the historical frame features in the memory Perform weighted summation to obtain the retrieval features, and add the retrieval features and query features The splicing is used as the final matching feature.

[0030] Furthermore, the step S17 is specifically as follows: using multiple residual blocks as decoders, taking the matching features in the step S16 and the query encoding features in the step S12 introduced by the jump connection as input, and outputting the final segmentation result M t .

[0031] Furthermore, the model inference of step 2 includes the following steps: S21, initializing the segmentation target, a mask of the target to be segmented is given in the first frame of the new video sequence, and the segmentation starts from the second frame of the video sequence;

[0032] S22, the current frame image I t Get the current frame key-value encoding pair by querying the encoding unit

[0033] S23, the previous frame image I t-1 and its target mask prediction result M t-1 Get the key-value coding pair of the previous frame through the reference coding unit The previous frame key-value coding pair temporarily stored in storage;

[0034] S24, passing the features output by the image branch of the reference coding unit through the state estimation unit to obtain the target mask prediction result M t-1 The state estimation score s t-1 ;

[0035] S25, executing the matching unit to obtain matching features;

[0036] S26: Input the matching features in step S25 and the query encoding features in step S22 into the decoding unit to obtain the target mask prediction result M of the current frame. t ;

[0037] S27, execute the memory management unit to organize the key-value coding pairs from the first frame to the previous frame of the image stored in the memory, so that the size of the storage space used is fixed at a given equivalent size, thereby improving the space utilization efficiency and the segmentation speed of specific targets (i.e., the targets given in the first frame) in subsequent images.

[0038] Furthermore, the step S27 is specifically as follows: the maximum value of the equivalent capacity of MB |MB| is set to M (at most M frames of image features are stored in the memory, M>1); the features of the first frame are placed in the memory MB by default, and when the equivalent capacity of the memory |MB| is less than M, the features temporarily stored in the step S23 are stored in the memory MB every 5 frames (time t-1 is a multiple of 5). and the state estimation score s in step S24 t-1 Retained in the memory, otherwise removed and discarded; when the equivalent capacity of the memory |MB| = M, if the feature The corresponding state score s t-1 <s th (s th is the state estimation threshold, with a value of 0.85), then it will be removed and discarded, otherwise the following merging operation will be performed: feature With the minimum state fraction s in the memory min Features Merge, state estimation score s t-1 With the minimum state score s min Merger; the merging process is:

[0039]

[0040]

[0041]

[0042] in, are the merged feature and state scores respectively.

[0043] Furthermore, the s in step S27 th The values ​​of and M can be customized and adjusted according to actual conditions.

[0044] Compared with the prior art, the invention has the following beneficial effects: a fast real-time video target segmentation method based on bimodal interaction and state feedback, which enhances the learning ability of the target appearance and improves the anti-interference ability and robustness of the segmentation process by realizing bimodal interaction of image features and mask features; provides the system with the state value of the segmentation result through state feedback, helps to better organize the historical frame features in the memory, and improves the inference speed and memory usage efficiency of the system. This method can accurately and quickly segment targets in many difficult practical scenarios, and has good results. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a schematic diagram of the network framework of a fast real-time video object segmentation method based on bimodal interaction and state feedback;

[0046] Figure 2 This is a fast real-time video target segmentation method based on bimodal interaction and state feedback and the effect diagram of two other existing methods. DETAILED DESCRIPTION

[0047] In order to better understand the above technical scheme, the technical scheme of the present invention is further described below by specific embodiments in combination with the accompanying drawings, so that the technical scheme is clearer and easier to understand. Through the content disclosed in this specification, those skilled in the art can easily understand other advantages and effects of the present invention. The present invention can also be implemented or applied with different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and the features in the embodiments can be combined with each other without conflict.

[0048] This embodiment relates to a fast real-time video target segmentation method based on bimodal interaction and state feedback, which mainly includes the steps of model training and model inference in two stages:

[0049] Step 1: In the model training phase, the network model is trained using a server with high-performance computing, and the network parameters are optimized by reducing the network loss function until the network converges, thereby obtaining the network weights of the fast real-time video target segmentation method based on bimodal interaction and state feedback;

[0050] Step 2: In the model inference phase, the network weights obtained in the training phase are used to segment the target given in the first frame in the new video sequence to achieve fast real-time target segmentation.

[0051] Figure 1 The invention discloses a network framework diagram of a fast real-time video target segmentation method based on bimodal interaction and state feedback; wherein, the query encoding unit takes the current frame image as input; the reference encoding unit is composed of an image branch and a mask branch, the image branch takes the previous frame image as input, and the mask branch takes the previous frame target mask prediction result as input; the mask reconstruction unit reconstructs the previous frame target mask prediction result; the state evaluation unit predicts the intersection-over-union ratio between the previous frame target mask prediction result and the true result as the predicted state value; the memory management unit organizes the historical frame features in the memory; the matching unit retrieves the historical frame information from the memory; the decoding unit takes the matching features and the features from the query encoding unit as input, and outputs the current frame target mask prediction result.

[0052] Figure 2This is a diagram showing the effect of a fast real-time video target segmentation method based on bimodal interaction and state feedback and two other existing methods. The first row shows the true value, the second row shows the result of our fast real-time video target segmentation method, the third row shows the result of the STM-cycle method, and the fourth row shows the result of the SiamMask method. From the comparison of the results, it can be seen that our method can still achieve good segmentation results in complex situations such as fast target movement, occlusion, appearance change, and background interference. For example, in the first and third columns of figures, when the target is occluded, other methods either segment the wrong target (STM-cycle) or the segmented target mask contour is relatively rough (SiamMask), and the results are not accurate, but our method can still accurately and effectively segment multiple targets. In the second column of figures, one of the targets (a smaller animal dog) has a large appearance change, and our method can correctly segment the corresponding target, while other methods either segment the target incorrectly (STM-cycle) or the segmentation is incomplete (SiamMask).

[0053] like Figure 1 As shown, the server of this embodiment includes multiple functional modules such as a query encoding unit, a reference encoding unit, a mask reconstruction unit, a state evaluation unit, a memory management unit, a matching unit, and a decoding unit. In addition, it also includes a training video segment generation unit. Specifically, each functional unit is as follows:

[0054] (a) a training video generation unit; used to obtain a large number of annotated video clips required for model training from any video or static image datasets of multiple video datasets;

[0055] (b) Query encoding unit; used to extract the current frame image I t The features of the query are generated into key-value pairs

[0056] (c) Reference coding unit; used to extract the previous frame image I t-1 and its target mask prediction result M t-1 The features of the reference frame are generated by the interaction between them. The reference encoding unit is designed with two branches to encode image features and mask features respectively, and a fusion model is used to inject low-level mask features into the image branch to enhance the learning of image features on target appearance and texture.

[0057] (d) a mask reconstruction unit; used to reconstruct the target mask prediction result of the previous frame input to the reference coding unit, so that the bimodal interaction process of the reference coding unit is more sufficient;

[0058] (e) a state evaluation unit; used to predict the intersection-over-union ratio of the target mask prediction result of the previous frame input to the reference encoding unit and the actual result as a state prediction value, provide state feedback for the system, and provide a reference basis for the feature organization process of the memory management unit;

[0059] (f) Memory management unit: used to organize the features in the memory so that the system can adaptively update the features in the memory, thereby improving the inference speed and memory utilization efficiency of the system;

[0060] (g) Matching unit: used to retrieve information from historical frame features in the memory. The matching unit matches the historical frame features in the memory according to the reference frame features, and can retrieve richer spatiotemporal information and learn a more accurate and stable target appearance;

[0061] (h) A decoding unit; used to obtain the final current frame target mask prediction result. The decoding unit generates the current frame target mask prediction result based on the matching features and the features from the query encoder.

[0062] The specific operation process of the model training phase in step 1 of the fast real-time video target segmentation method based on dual-modal interaction and state feedback of this embodiment is as follows:

[0063] Step S11, executing the training video segment generating unit to generate a training video segment of length T, where T ≥ 2; specifically, randomly extracting T images at intervals from any video in the multiple video data sets, and performing T different affine transformations (combinations of translation, scaling, flipping, rotation, and shearing) on ​​the T images to form a training video segment; or, randomly extracting an image from the image data set, and performing T different affine transformations to form a training video segment;

[0064] Step S12: Execute the query encoding unit to query the image (current frame image I t ) key-value coding pair, the query image key-value coding pair is defined as The subscript t represents the time (t>1), Q represents the query image, and k, v corresponding to the Query frame; specifically, a deep residual network is used to extract the input query frame image features, and then the extracted features are input into two parallel convolutional layers to obtain the key-value coding pair

[0065] Step S13: Execute the reference encoding unit to extract the reference frame image (previous frame image I t-1 ) features and reference frame target mask (previous frame target mask prediction result M t-1 ) features and perform bimodal interaction, outputting reference key-value coding pairs Specifically, a two-branch residual network (image branch and mask branch) without shared parameters is first used to extract features from the input reference frame image and the reference frame target mask prediction result respectively; then the features of each stage of the mask branch and the features of the corresponding stage of the image branch are added through a squeeze excitation block (SE block); then the added features are injected into the image branch; finally, the image branch outputs the reference key-value coding pair Reference key-value coding pairs is directly stored in the memory MB, R represents the reference image, and k,v,corresponding to the Reference frame;

[0066] Step S14, executing a mask reconstruction unit to reconstruct the target mask prediction result of the mask branch input into the step S13, so that the mask branch can learn richer target spatial features; specifically, using a spatial pyramid structure (ASPP) with a dilated convolution and a convolutional layer to form a segmentation head, placing it behind the mask branch, taking the output of the mask branch as input, and outputting a reconstructed target mask;

[0067] Step S15, executing the state estimation unit, performing state estimation on the target mask prediction result input in the step S13, and providing state feedback; specifically, using a state evaluator consisting of two 3×3 convolutions, a global average pooling, a 1×1 convolution, and a sigmoid normalization operation to predict the intersection over union (IoU) between the target mask prediction result and the true result; the state evaluator is placed after the image branch in the step S13, takes the output of the image branch as input, and outputs the predicted state score s t-1 ;

[0068] Step S16: Execute the matching unit to retrieve information from the historical frame features in the memory according to the query frame features; specifically, use the query image key-value encoding to In The historical frame features in the memory Calculate the similarity and normalize it as a weight for the historical frame features in the memory Perform weighted summation to obtain the retrieval features, and add the retrieval features and query features Splicing is used as the final matching feature;

[0069] Step S17: Execute the decoding unit to output the query frame (current frame I t ) Segmentation result M t Specifically, multiple residual blocks are used as decoders, the matching features in step S16 and the query encoding features in step S12 introduced by the jump connection are used as input, and the final segmentation result M is output. t ;

[0070] Step S18: Perform network training in an end-to-end manner; specifically, the segmentation loss function L s , reconstruction loss function L r And the state evaluation loss function L e Combine and use adaptive weights to automatically balance the loss functions of each part to obtain the final loss function L total ; The mathematical expression of each loss is:

[0071] L s (Y t ,M t )=L ce (Y t ,M t )+α·L IoU (Y t ,M t )

[0072] L r (Y t-1 ,M r,t-1 )=L ce (Y t-1 ,M r,t-1 )+α·L IoU (Y t-1 ,M r,t-1 )

[0073]

[0074]

[0075] in, represents the cross entropy loss;

[0076] represents Mask IoU loss; Y t-1 ,Y t Respectively represent the target mask truth value of the previous frame and the target mask truth value of the current frame; M t-1 ,M t Respectively represent the target mask prediction result of the previous frame and the target mask prediction result of the current frame; M r,t-1 Represents the reconstructed value of the target mask prediction result of the previous frame; s t-1 Indicates M t-1 The state prediction score of ; Ω represents the set of all pixels in the target mask; T represents the length of the training video segment (the value is 3, which can be flexibly adjusted according to the computing resources); α is a hyperparameter (the value is 1); w1, w2 are learnable parameters;

[0077] Step S19: Optimize the objective function to obtain the local optimal network parameters; specifically, the total loss function L in step S18 istotal As the objective function, the Adam optimizer is used to iteratively update the network parameters to reduce the objective loss function until it converges to the local optimum. At this point, the training is completed and the trained network weights for fast real-time video target segmentation based on bimodal fusion and state feedback are obtained.

[0078] The specific steps of the inference phase of step 2 in this embodiment are as follows:

[0079] Step S21, initializing the segmentation target, the mask of the target to be segmented will be given in the first frame of the new video sequence, and the segmentation starts from the second frame of the video sequence;

[0080] Step S22: The current frame image (I t ) Obtain the current frame key-value encoding pair through the query encoding unit in step S12

[0081] Step S23: The previous frame image (I t-1 ) and its target mask prediction result (M t-1 ) Obtain the key-value coding pair of the previous frame through the reference coding unit in step S13 is temporarily placed in storage MB;

[0082] Step S24: The feature output by the image branch of the reference encoding unit in step S13 is used by the state estimation unit in step S15 to obtain the target mask prediction result (M t-1 )’s state estimation score s t-1 ;

[0083] Step S25, executing the matching unit in step S16 to obtain matching features;

[0084] Step S26: Input the matching features in step S25 and the query encoding features in step S22 into the decoding unit in step S17 to obtain the target mask prediction result M of the current frame. t ;

[0085] Step S27, execute the memory management unit to organize the historical frame features in the memory MB, so that the size of the used storage space is fixed and the subsequent inference speed is improved; specifically, set the maximum value of the equivalent capacity |MB| of MB to M (at most M frames of image features are stored in the memory, M>1, which can be adjusted according to the inference speed and space requirements); the features of the first frame are placed in the memory MB by default, and when the equivalent capacity |MB| of the memory is less than M, every 5 frames (time t-1 is a multiple of 5), the features temporarily stored in step S23 and the state estimation score s in step S24 t-1Retained in the memory, otherwise removed and discarded; when the equivalent capacity of the memory |MB| = M, if the feature The corresponding state score s t-1 <s th (s th is the state estimation threshold, the value is 0.85, which can be adjusted according to the actual situation), then it will be removed and discarded, otherwise the following merging operation will be performed: feature With the minimum state fraction s in the memory min Features Merge, state estimation score s t-1 With the minimum state score s min Merger; the merging process is:

[0086]

[0087]

[0088]

[0089] in, are the merged feature and state scores respectively.

[0090] The above embodiments are the best implementation methods of the present invention, but the implementation methods of the present invention are not limited by the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principle of the present invention shall be equivalent replacement methods and shall be included in the protection scope of this aspect. More specifically, within the scope of the disclosure, drawings and claims of this application, various modifications and improvements may be made to the components and / or layout of the subject combination layout. In addition to the modifications and improvements made to the components and / or layout, other uses will also be obvious to those skilled in the art.

Claims

1. A fast real-time video target segmentation method based on dual-modal interaction and state feedback, characterized in that: The following steps are involved: Step 1: Model training: Use the server to train the network model, optimize the network parameters by reducing the network loss function until the network converges, and obtain the network weights for fast real-time video target segmentation based on bimodal interaction and state feedback; Step 2: Model inference: Use the network weights obtained in the training phase to segment the given target in the first frame of the new video sequence; The model training of step 1 specifically includes the following steps: S11, using the server to execute a training video segment generating unit to generate a training video segment of length T, where T ≥ 2; S12, using the server to execute the query encoding unit to extract the query image key-value encoding pair, the current frame image is I t , the query image key-value encoding pair is t represents the time, t>1, Q represents the query image; S13, using the server to execute the reference encoding unit, using the dual-branch residual network with unshared parameters of the protection image branch and the mask branch, extracting the reference frame image features and the reference frame target mask features and performing dual-modal interaction, the previous frame image of the reference frame image is I t-1 , the target mask prediction result of the previous frame of the reference frame target mask is M t-1 , output reference key-value coding pairs and stored in a memory, R represents a reference image, and M refers to the maximum value of the equivalent capacity of the memory; S14, using the server to execute a mask reconstruction unit to reconstruct the target mask prediction result of the mask branch input into step S13, and output the reconstructed target mask; S15, using the server to execute the state estimation unit, perform state estimation on the target mask prediction result input in step S13, provide state feedback, and output the predicted state score s t-1 ; S16, using the server to execute the matching unit, according to the query image key-value encoding To retrieve the information in the historical frame features in the memory and obtain the final matching features; S17, using the server to execute the decoding unit and output the final segmentation result M of the query frame t ; S18, using the server to perform network training in an end-to-end manner; specifically, the segmentation loss function L s , reconstruction loss function L r And the state evaluation loss function L e Combine and use adaptive weights to automatically balance the loss functions of each part to obtain the final total loss function L total ; S19, using the server to optimize the objective function, the objective function adopts the total loss function L in step S18 total , obtaining the local optimal network parameters as the network weights for fast real-time video target segmentation based on bimodal fusion and state feedback; the model inference of step 2 includes the following steps: S21, initializing the segmentation target, the mask of the target to be segmented will be given in the first frame of the new video sequence, and the segmentation starts from the second frame of the video sequence; S22, the current frame image I t Get the current frame key-value encoding pair by querying the encoding unit S23, the previous frame image I t-1 and its target mask prediction result M t-1 Get the key-value coding pair of the previous frame through the reference coding unit The previous frame key-value coding pair temporarily stored in storage; S24, passing the features output by the image branch of the reference coding unit through the state estimation unit to obtain the target mask prediction result M t-1 The state estimation score s t-1 ; S25, executing the matching unit to obtain matching features; S26: Input the matching features in step S25 and the query encoding features in step S22 into the decoding unit to obtain the target mask prediction result M of the current frame. t ; S27, executing the memory management unit to organize the key-value coding pairs from the first frame to the previous frame of the image stored in the memory, so that the size of the used storage space is fixed at a given equivalent size, thereby improving the space utilization efficiency and the segmentation speed of the target given in the first frame in the subsequent images; The step S27 specifically includes: setting the maximum value of the equivalent capacity |MB| of the memory MB to M, and the memory can store features of at most M frames of images, M>1; and by default, the features of the first frame are stored in the memory; When the equivalent capacity of the memory |MB|<M, every 5 frames, the time t-1 is a multiple of 5, and the feature temporarily stored in step S23 and the state estimation score s in step S24 t-1 Retained in storage, otherwise removed and discarded; When the equivalent capacity of the memory |MB| = M, if the characteristic The corresponding state score s t-1 <s th , it is removed and discarded, otherwise it is merged: Features With the minimum state fraction s in the memory m i n Features Merge, state estimation score s t-1 With the minimum state score s m i n merge, in, are the merged feature and state scores respectively.

2. According to claim 1, a fast real-time video target segmentation method based on dual-modal interaction and state feedback is characterized in that: The step S11 specifically includes: randomly extracting T images at intervals from any video of the multiple video data sets, and performing T different affine transformations on the T images respectively, where the affine transformations include translation, scaling, flipping, rotation and shearing, to form training video clips; Alternatively, an image is randomly selected from the image dataset and subjected to T different affine transformations to form a training video clip.

3. The fast real-time video target segmentation method based on dual-modal interaction and state feedback according to claim 1, characterized in that: The step S13 specifically comprises: using a dual-branch residual network including an image branch and a mask branch that do not share parameters to extract features from the input reference frame image and the reference frame target mask prediction result respectively; then adding the features of each stage of the mask branch and the features of the corresponding stage of the image branch through a squeeze excitation block respectively; Then the added features are injected into the image branch; finally, the image branch outputs the reference key-value coding pair Reference key-value coding pairs are stored directly into memory.

4. The fast real-time video target segmentation method based on dual-modal interaction and state feedback according to claim 1, characterized in that: The step S16 specifically includes: using the query image key-value encoding In The historical frame features in the memory Calculate the similarity and normalize it as a weight for the historical frame features in the memory Perform weighted summation to obtain the retrieval features, and add the retrieval features and query features The splicing is used as the final matching feature.

5. The fast real-time video target segmentation method based on dual-modal interaction and state feedback according to claim 4 is characterized in that: The step S17 is specifically as follows: using multiple residual blocks as decoders, taking the matching features in the step S16 and the query encoding features in the step S12 introduced by the skip connection as input, and outputting the final segmentation result M t .

6. The fast real-time video target segmentation method based on dual-modal interaction and state feedback according to claim 1, characterized in that: The state estimation threshold s in step S27 th Use 0.85 or customize; alternatively, the value of the maximum value M of the equivalent capacity of the memory in step S27 can be customized.

Images