Semi-supervised video object segmentation method and device based on query mechanism

Abstract

The application discloses a semi-supervised video object segmentation method based on a Query mechanism, which comprises the following steps: inputting a reference frame and a current frame into the same backbone network respectively to extract image features; extracting mask features of a segmentation mask of the reference frame through a convolution layer, then adding the mask features and corresponding image features to obtain reference features; using the reference features of the reference frame as a reference to construct a key-value pair and input the key-value pair into a Transformer, and inputting foreground Query and background Query as probes into the Transformer; using the foreground Query and the background Query as a reference, and inputting image features of the current frame as probes into another Transformer; inputting the image features of the current frame into a feature pyramid network for decoding; performing matrix multiplication on the foreground Query, the background Query and the decoded current frame features to perform fusion; taking the maximum of each pixel in the same group of Query, performing a pixel-by-pixel Softmax operation on the foreground mask and the background mask, and judging whether it is foreground or background according to the value size. The application also provides a corresponding semi-supervised video object segmentation device based on the Query mechanism.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of deep learning and computer vision, and more particularly relates to a semi-supervised video object segmentation method and device based on a Query mechanism. BACKGROUND

[0002] Semi-supervised video object segmentation is one of the important tasks in the field of computer vision, and also has a wide range of applications in the fields of video processing, automatic driving, etc. The algorithm needs to track and pixel-level locate the target object in the subsequent frames according to the annotation of the first frame of the given video, and has a high requirement for the accuracy of pixel-level location. The previous mainstream semi-supervised video object segmentation method mainly follows two paradigms: 1) detection-based paradigm: first, use an advanced object detection algorithm to detect objects in each frame, then extract features for each detected object, and perform feature matching with the target object in the first frame, the object that successfully matches is considered as the tracked target object, and then performs object segmentation (pixel-level location) according to the previous features; 2) pixel matching-based paradigm: extract features for each frame, and perform pixel-level matching between the current frame and the reference frame, and determine whether each pixel in the current frame belongs to the target object according to the features obtained during matching. The detection-based paradigm requires the use of an additional object detection network, and can only track the object categories that the object detection network can detect, and is not universal. The pixel-level matching step in the pixel-level matching-based paradigm has high time and space complexity, especially when processing high-resolution videos or long videos, it is easy to cause excessive computation and unable to process videos or slow processing speed. SUMMARY

[0003] In view of the above defects or improvement needs of the prior art, the present application provides a semi-supervised video object segmentation scheme based on a Query mechanism, which is faster when processing large resolution videos, has stronger scalability, and has a simpler structure.

[0004] To achieve the above purpose, according to one aspect of the present application, a semi-supervised video object segmentation method based on a Query mechanism is provided, comprising the following steps:

[0005] Step one: input the reference frame and the current frame into the same backbone network respectively to extract image features, wherein the reference frame includes the first frame with a real segmentation mask and the previous frame of the current frame;

[0006] Step two: extract mask features of the reference frame through a convolution layer, and then add and fuse the corresponding image features to obtain reference features;

[0007] Step three: use the reference features of the reference frame as a reference to build a key-value pair input Transformer, and at the same time, the foreground Query and the background Query are used as probe input Transformer, so as to compress the information in the reference features into the foreground Query and the background Query;

[0008] Step four: use the foreground Query and the background Query obtained in the last step as a reference, and the image features of the current frame as a probe to input another Transformer, so as to transmit the information in the Query into the image features of the current frame;

[0009] Step five: input the image features of the current frame into a feature pyramid network for decoding to increase the resolution;

[0010] Step six: multiply the foreground Query and the background Query with the decoded current frame features to obtain a segmentation mask corresponding to each Query;

[0011] Step seven: the Queries in the same group take the maximum value for each pixel to obtain two masks of foreground and background, and then perform a pixel-by-pixel Softmax operation on the two masks to determine whether it is foreground or background according to the value.

[0012] In an embodiment of the present application, a Query propagation mechanism from the front to the back of the video is used, specifically, the values of the input foreground Query and the background Query are changed.

[0013] In an embodiment of the present application, the values of the input foreground Query and the background Query are changed, specifically:

[0014] For the first frame that needs to be predicted in a video, the Query value learned during training is used to initialize the Query, and then the image of the first frame of the video and the object information contained in the mask are compressed into the Query through the Transformer in step three.

[0015] In an embodiment of the present application, the values of the input foreground Query and the background Query are changed, specifically:

[0016] For the frames after the first frame that need to be predicted in the video, the foreground Query and the background Query output in step three when processing the last frame are directly used as the input Query of the current frame, so that the information in the Query can be transmitted from the first frame to the last frame in the video.

[0017] In one embodiment of the present application, the method further comprises introducing a spatio-temporal embedding feature, and enabling the Transformer to perceive the spatio-temporal information by adding the spatio-temporal embedding feature to the image feature input in the Transformer before step three.

[0018] In one embodiment of the present application, a spatial embedding feature is added to the current image feature before step four.

[0019] In one embodiment of the present application, a spatial embedding feature is added to the input of each layer of the feature pyramid network of step five.

[0020] In one embodiment of the present application, the first frame to be predicted is the second frame of the video.

[0021] According to another aspect of the present application, there is also provided a semi-supervised video object segmentation device based on a Query mechanism, comprising at least one processor and a memory, which are connected through a data bus, the memory stores instructions executable by the at least one processor, and the instructions are used to complete the semi-supervised video object segmentation method based on the Query mechanism after being executed by the processor.

[0022] Overall, compared with the prior art, the above technical solutions conceived by the present application have the following beneficial effects:

[0023] (1) The present application proposes a new Query-based paradigm to solve the problems of large amount of calculation, incompatibility with the network structure of other tasks, etc. caused by the previous mainstream semi-supervised video object segmentation method based on pixel matching, which is faster in processing large resolution videos, more scalable, and simpler in structure.

[0024] (2) The present application proposes a semi-supervised video object segmentation method based on a Query mechanism, which is simpler in structure, faster, more scalable, and can naturally track and segment multiple targets at the same time, compared with the previous mainstream matching-based method, and can effectively replace other semi-supervised object segmentation methods. Meanwhile, based on this new architecture, the present application designs a unique Query propagation mechanism according to the task characteristics of semi-supervised video object segmentation. Ultimately, this architecture is efficient and accurate, and achieves the state-of-the-art speed and accuracy on multiple datasets.

[0025] (3) The application provides a novel Query propagation mechanism, which can effectively propagate high-quality mask information of a first frame in the processing of subsequent frames, thereby improving the model effect. Meanwhile, considering that updating and refining the Query features is not as difficult as compressing object information into the Query from zero when processing the first frame, in order to further improve the algorithm speed, for these subsequent frames, the application only inputs the Query of the previous frame into the last few units of the Transformer, thereby reducing the calculation amount and improving the speed. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is an overview diagram of the Query-based semi-supervised video object segmentation paradigm in the application;

[0027] Figure 2 is an overview diagram of the Query-based semi-supervised video object segmentation network structure in the application;

[0028] Figure 3 is a flow diagram of the Query-based semi-supervised video object segmentation method in the application. DETAILED DESCRIPTION

[0029] In order to make the objectives, technical solutions and advantages of the application clearer, the application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the application and do not limit the application. In addition, the technical features involved in each embodiment of the application described below can be combined with each other as long as they do not conflict with each other.

[0030] As shown in Figure 1 , it is an overview diagram of the Query-based semi-supervised video object segmentation paradigm. The Query-based semi-supervised object segmentation paradigm embeds object information into foreground Query and background Query. The Query is propagated from the first frame to the last frame of the video, is constantly updated, and uses the object information therein to track and segment objects.

[0031] As shown in Figure 2 , it is an overview diagram of the Query-based semi-supervised video object segmentation network structure provided by the application. In addition to the backbone network for extracting features, it can be divided into three modules: a foreground Query and background Query perception module for embedding object information into the Query; a pixel-level feature module for obtaining detailed pixel-level features according to the Query and the current image features; and a segmentation module for completing the final segmentation according to the Query and the pixel-level features.

[0032] As shown in Figure 3As shown, the application provides a semi-supervised video object segmentation method based on the Query mechanism, comprising the following steps:

[0033] Step one: input the reference frame and the current frame into the same backbone network to extract image features. The reference frame includes the first frame with a real segmentation mask and the previous frame of the current frame;

[0034] Step two: extract the mask features of the reference frame through a convolution layer, then add and fuse the corresponding image features to obtain the reference features;

[0035] Step three: use the reference features of the reference frame as a reference to construct a key-value pair input into the Transformer, and the foreground Query and the background Query as probes to input into the Transformer, thereby compressing the information in the reference features into two groups (foreground and background) Query; at the same time, according to the learned attention weight, the foreground Query mainly focuses on the foreground information, and the background Query mainly focuses on the background information;

[0036] Step four: input the foreground Query and the background Query obtained in the above step as a reference, and the image features of the current frame as a probe into another Transformer, thereby transmitting the information in the Query into the image features of the current frame;

[0037] Step five: input the image features of the current frame into the feature pyramid network for decoding to increase the resolution;

[0038] Step six: multiply the foreground Query and the background Query with the decoded current frame features to obtain a segmentation mask corresponding to each Query;

[0039] Step seven: take the maximum of each pixel in the same group of Query to obtain two masks of foreground and background, and then perform a pixel-by-pixel Softmax operation on the two masks to determine whether it is foreground or background according to the value.

[0040] In order to play the advantage of the Query mechanism and fully utilize the time sequence information of the video, the application further proposes a Query propagation mechanism from front to back along the video. Specifically, the values of the input foreground Query and background Query can be changed, which can be divided into the following two cases:

[0041] Case one: for the first frame in a video that needs to be predicted (the second frame of the whole video, because the annotation of the first frame is given and does not need to be predicted), the application uses the Query value learned during training to initialize Query; then the image of the first frame of the video and the object information contained in the mask are compressed into Query through the Transformer described in step three above;

[0042] Case two: for the frames after the first frame in the video, the application can directly use the foreground Query and background Query output in step three when processing the previous frame as the input Query of the current frame, so that the information in Query can be passed from the first frame to the last frame in the video. In this process, the information in Query is constantly refined and improved, and at the same time is constantly updated to introduce new information of the current frame. Since the mask information of the first frame in the semi-supervised video object segmentation task is manually annotated, the quality of the mask will generally decrease as the object moves in the video. Through this novel Query propagation mechanism in the application, the high-quality mask information of the first frame can be effectively propagated step by step in the processing of the subsequent frames, thereby improving the model effect. At the same time, considering that updating and refining the Query features is not as difficult as compressing the object information into Query from zero when processing the first frame, in order to further improve the algorithm speed, for these subsequent frames, the application only inputs the Query of the previous frame into the last few units of the Transformer, thereby reducing the amount of calculation and improving the speed.

[0043] In addition, in order to enable the algorithm to perceive the time and space information in the video and make good use of the similarity of object positions between adjacent frames, the application additionally introduces spatio-temporal embedding features, which are added to the image feature input in the Transformer before step three, so that the Transformer can perceive the spatio-temporal information. At the same time, the application also adds spatial embedding features to the current image features before step four, and adds spatial embedding features to the input of each layer of the feature pyramid network in step five, in order to improve the model effect.

[0044] Using the semi-supervised video object segmentation method based on Query in the application, high precision and speed can be achieved in commonly used public data sets, and the performance is better than most methods. The semi-supervised video object segmentation evaluation index J&F on the validation set of the DAVIS2017 data set can reach 83.8, while maintaining a relatively fast speed of 22.9FPS, which is better than most methods based on other paradigms and comparable to the current top performance.

[0045] Further, the application further provides a semi-supervised video object segmentation device based on a Query mechanism, comprising at least one processor and a memory, the at least one processor and the memory are connected through a data bus, the memory stores instructions executable by the at least one processor, and the instructions are used to complete the semi-supervised video object segmentation method based on the Query mechanism after being executed by the processor.

[0046] Those skilled in the art can easily understand that the above description is only the preferred embodiment of the present application, and is not used to limit the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A semi-supervised video object segmentation method based on Query mechanism, characterized in that, Comprising the following steps: Step one: input the reference frame and the current frame into the same backbone network to extract image features, wherein the reference frame includes the first frame with a given true segmentation mask and the previous frame of the current frame; Step two: extract the mask feature of the reference frame through a convolution layer, then add it to the corresponding image feature for fusion to obtain the reference feature; Step three: use the reference feature of the reference frame as a reference to construct a key-value pair input into the Transformer, and input the foreground Query and the background Query as probes into the Transformer, thereby compressing the information in the reference feature into the foreground Query and the background Query; Step four: input the foreground Query and the background Query obtained in the previous step as a reference, and input the image feature of the current frame as a probe into another Transformer, thereby transmitting the information in the Query into the image feature of the current frame; Step five: input the image feature of the current frame into a feature pyramid network for decoding to increase the resolution; Step six: multiply the foreground Query and the background Query with the decoded current frame feature to obtain a segmentation mask corresponding to each Query; Step seven: take the maximum value of each pixel in the same group to obtain two masks of foreground and background, and then perform a pixel-by-pixel Softmax operation on the two masks to determine whether it is foreground or background according to the value.

2. The semi-supervised video object segmentation method based on Query mechanism according to claim 1, wherein, Adopt a Query propagation mechanism from the front to the back of the video, specifically by changing the values of the input foreground Query and the background Query.

3. The semi-supervised video object segmentation method based on Query mechanism according to claim 2, wherein, Change the values of the input foreground Query and the background Query, specifically: For the first frame that needs to be predicted in a video, use the learned Query value during training to initialize the Query, and then compress the image and mask information of the first frame of the video into the Query through the Transformer in step three.

4. The semi-supervised video object segmentation method based on Query mechanism according to claim 2, wherein, Change the values of the input foreground Query and the background Query, specifically: For the frames after the first frame that need to be predicted in the video, directly use the foreground Query and the background Query output in step three when processing the previous frame as the input Query of the current frame, so that the information in the Query can be transmitted from the first frame to the last frame in the video.

5. The semi-supervised video object segmentation method based on Query mechanism according to claim 1 or 2, characterized in that, Further comprising: Introduce a spatiotemporal embedding feature, which is added to the image feature input into the Transformer in step three to enable the Transformer to perceive spatiotemporal information.

6. The semi-supervised video object segmentation method based on Query mechanism according to claim 1 or 2, characterized in that, Add a spatial embedding feature to the current image feature before step four.

7. The semi-supervised video object segmentation method based on Query mechanism according to claim 1 or 2, characterized in that, Add a spatial embedding feature to the input of each layer of the feature pyramid network in step five.

8. The semi-supervised video object segmentation method based on Query mechanism according to claim 3, wherein, The first frame that needs to be predicted is the second frame of the video.

9. A semi-supervised video object segmentation device based on a Query mechanism, characterized by: The application relates to a device comprising at least one processor and a memory, which are connected through a data bus, the memory storing instructions executable by the at least one processor, the instructions being used to complete the semi-supervised video object segmentation method based on the Query mechanism according to any one of claims 1-8 after being executed by the processor.

Images