Video object segmentation method, device, equipment and medium

Abstract

The present application discloses a video target segmentation method, device, equipment and medium, which relates to the field of artificial intelligence. The method comprises: extracting features from support images in a support set to obtain target object features of a target object in the support image; extracting features from a video image in a query video to obtain video image features of the video image; determining a first attention matrix and a second attention matrix based on the target object features and the video image features, wherein the first attention matrix is ​​an attention matrix between the support set and a domain agent, and the second attention matrix is ​​an attention matrix between the query video and the domain agent; determining a target attention feature based on the first attention matrix and the second attention matrix, wherein the target attention feature is used to represent the attention of the support set on the query video; performing target segmentation based on the target attention feature to obtain a target segmentation result, wherein the target segmentation result is used to characterize the position of the target object in each frame of the video image.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence, and in particular to a video target segmentation method, device, equipment and medium. Background Art

[0002] Few-shot video object segmentation is a technique that uses a small number of annotated supporting images as cues to segment specific objects from video images, where the annotations of the supporting images are used to indicate the locations of specific objects in the supporting images.

[0003] In related technologies, a fully connected many-to-many attention mechanism is usually used to achieve few-sample video target segmentation. When performing target segmentation, the computer device extracts features from the support image and the video image respectively, and uses the extracted features as the input of the attention model, which extracts attention features and finally achieves target segmentation based on the extracted attention features.

[0004] However, under the fully connected many-to-many attention mechanism, the amount of computation in the target segmentation process increases exponentially with the increase of supporting images and video images, resulting in a slower target segmentation speed and even the problem of being unable to complete target segmentation. Summary of the invention

[0005] The embodiments of the present application provide a method, device, equipment and medium for video object segmentation, which can reduce the amount of calculation in the object segmentation process, thereby increasing the speed of object segmentation. The technical solution is as follows:

[0006] On the one hand, an embodiment of the present application provides a video object segmentation method, the method comprising:

[0007] Performing feature extraction on the support image in the support set to obtain target object features of the target object in the support image;

[0008] Extracting features of the video image in the query video to obtain video image features of the video image;

[0009] Based on the target object features and the video image features, determining a first attention matrix and a second attention matrix, wherein the first attention matrix is ​​an attention matrix between the support set and the domain agent, and the second attention matrix is ​​an attention matrix between the query video and the domain agent, and the domain agent is a frame of video image in the query video;

[0010] Determine a target attention feature based on the first attention matrix and the second attention matrix, wherein the target attention feature is used to represent the attention of the support set on the query video;

[0011] Target segmentation is performed based on the target attention feature to obtain a target segmentation result, and the target segmentation result is used to characterize the position of the target object in the video image of each frame.

[0012] Optionally, the video image features are extracted by an encoder, and the target segmentation result is decoded by a decoder, and the encoder and the decoder are jump-connected.

[0013] On the other hand, an embodiment of the present application provides a video object segmentation method, the method comprising:

[0014] Performing feature extraction on a first sample support image in a first sample support set to obtain a first sample object feature of a first sample object in the first sample support image;

[0015] Extracting features of a sample video image in a sample query video to obtain sample video image features of the sample video image;

[0016] Input the first sample object feature and the sample video image feature into an attention model to obtain a sample attention feature, wherein the sample attention feature is used to represent the attention of the first sample support set on the sample query video, and the attention model is used to determine a first sample attention matrix and a second sample attention matrix based on the first sample target object feature and the sample video image feature, and determine the sample attention feature based on the first sample attention matrix and the second sample attention matrix, wherein the first sample attention matrix is ​​an attention matrix between the first sample support set and a sample domain agent, and the second sample attention matrix is ​​an attention matrix between the sample query video and the sample domain agent, and the sample domain agent is a frame of sample video image in the sample query video;

[0017] Based on the sample attention feature, target segmentation is performed by a decoder to obtain a sample segmentation result, wherein the sample segmentation result is used to characterize the position of the first sample object in each frame of the sample video image;

[0018] The attention model and the decoder are trained based on the sample segmentation result and the sample annotation information corresponding to the sample video image.

[0019] Optionally, based on the sample attention feature, target segmentation is performed by a decoder to obtain a sample segmentation result, including:

[0020] Perform feature concatenation on the sample attention feature and the second sample value feature to obtain the sample feature of the sample query video, wherein the second sample value feature is obtained by mapping the sample video image features;

[0021] The sample features are upsampled and convolved through a decoder to obtain a sample segmentation mask corresponding to each frame of the sample video image, wherein the sample segmentation mask is a binary mask, and the pixel value in the sample segmentation mask is used to indicate whether the corresponding pixel point belongs to the first sample object.

[0022] On the other hand, an embodiment of the present application provides a video object segmentation device, the device comprising:

[0023] A first extraction module is used to extract features from a support image in a support set to obtain target object features of a target object in the support image;

[0024] A second extraction module is used to extract features of the video image in the query video to obtain video image features of the video image;

[0025] A first matrix determination module is used to determine a first attention matrix and a second attention matrix based on the target object feature and the video image feature, wherein the first attention matrix is ​​an attention matrix between the support set and the domain agent, and the second attention matrix is ​​an attention matrix between the query video and the domain agent, and the domain agent is a frame of video image in the query video;

[0026] A first feature determination module, configured to determine a target attention feature based on the first attention matrix and the second attention matrix, wherein the target attention feature is used to represent the attention of the support set on the query video;

[0027] The first segmentation module is used to perform target segmentation based on the target attention feature to obtain a target segmentation result, and the target segmentation result is used to characterize the position of the target object in the video image of each frame.

[0028] On the other hand, an embodiment of the present application provides a video object segmentation device, the device comprising:

[0029] a third extraction module, configured to perform feature extraction on a first sample support image in a first sample support set to obtain a first sample object feature of a first sample object in the first sample support image;

[0030] A fourth extraction module, used for extracting features of a sample video image in a sample query video to obtain sample video image features of the sample video image;

[0031] A second feature determination module is configured to input the first sample object feature and the sample video image feature into an attention model to obtain a sample attention feature, wherein the sample attention feature is used to represent the attention of the first sample support set on the sample query video, and the attention model is configured to determine a first sample attention matrix and a second sample attention matrix based on the first sample target object feature and the sample video image feature, and determine the sample attention feature based on the first sample attention matrix and the second sample attention matrix, wherein the first sample attention matrix is ​​an attention matrix between the first sample support set and a sample domain agent, and the second sample attention matrix is ​​an attention matrix between the sample query video and the sample domain agent, and the sample domain agent is a frame of sample video image in the sample query video;

[0032] A second segmentation module is used to perform target segmentation through a decoder based on the sample attention feature to obtain a sample segmentation result, wherein the sample segmentation result is used to characterize the position of the first sample object in each frame of the sample video image;

[0033] The first training module is used to train the attention model and the decoder based on the sample segmentation result and the sample annotation information corresponding to the sample video image.

[0034] On the other hand, an embodiment of the present application provides a computer device, which includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the video target segmentation method as described in the above aspects.

[0035] On the other hand, an embodiment of the present application provides a computer-readable storage medium, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement the video object segmentation method as described in the above aspects.

[0036] On the other hand, an embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the video object segmentation method provided in the above aspect.

[0037] In an embodiment of the present application, feature extraction is performed on the support images in the support set and the video images in the query video respectively. After obtaining the target object features and the video image features, a frame of video images in the query video is used as a domain agent. Based on the target object features and the video image features, a first attention matrix between the support set and the domain agent and a second attention matrix between the query video and the domain agent are determined. Then, based on the first attention matrix and the second attention matrix, the target attention features of the support set acting on the query video are determined. Finally, the target attention features are used to perform target segmentation to obtain the position of the target object in each frame of the video image. By introducing the domain agent, the fully connected attention matrix between the support set and the query video under the many-to-many attention mechanism is decomposed into two many-to-one attention matrices bridged by the domain agent. Therefore, as the number of support images and video images increases, the growth mode of computational complexity and memory consumption in the target segmentation process changes from exponential growth to linear growth, thereby reducing the probability of being unable to perform target segmentation due to excessive computational complexity, thereby achieving the effect of increasing the target segmentation speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 It is a schematic diagram of the principle of the video target segmentation process in the related art;

[0040] Figure 2 Schematic diagram of the principle of the video target segmentation process in the embodiment of the present application;

[0041] Figure 3 A schematic diagram showing an implementation environment provided by an exemplary embodiment of the present application is shown;

[0042] Figure 4 A flowchart of a video object segmentation method provided by an exemplary embodiment of the present application is shown;

[0043] Figure 5 A flowchart of a video object segmentation method provided by another exemplary embodiment of the present application is shown;

[0044] Figure 6 is a schematic diagram of an implementation of a video object segmentation process shown in an exemplary embodiment of the present application;

[0045] Figure 7 A flowchart of a video object segmentation method provided by another exemplary embodiment of the present application is shown;

[0046] Figure 8 A flowchart of a video object segmentation method provided by another exemplary embodiment of the present application is shown;

[0047] Fig. 9 is a schematic diagram of an implementation of meta-learning and online learning process shown in an exemplary embodiment of the present application;

[0048] Fig.10 This is a comparison chart of memory consumption during segmentation under different schemes;

[0049] Fig.11 This is a comparison chart of the time consumption for segmentation under different schemes;

[0050] Fig.12 This is a comparison chart of the target segmentation effects under different schemes;

[0051] Fig.13 is a structural block diagram of a video object segmentation device provided by an exemplary embodiment of the present application;

[0052] Fig.14 is a structural block diagram of a video object segmentation device provided by an exemplary embodiment of the present application;

[0053] Fig.15 A schematic diagram of the structure of a computer device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0054] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0055] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0056] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0057] Computer vision technology (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to replace human eyes to identify, track and measure targets, and further perform graphics processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology generally includes image processing, image recognition, image segmentation, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition. The video target segmentation method involved in the embodiment of the present application, that is, the application of computer vision technology in the field of image segmentation, can achieve the segmentation of target objects in the video using only a small number of supporting images (i.e., a small number of samples).

[0058] like Figure 1 As shown in FIG. 1 , under the many-to-many attention mechanism, the video target segmentation based on a few samples can be modeled as follows: the support images in the support set 11 and the video images in the query video 12 are used as two sets of vertices of the bipartite graph, and the similarity between the vertices on both sides is used as the connection weight through the fully connected bipartite graph. Based on the connection weight, the features of the corresponding vertices of each support image connected to the corresponding vertex of the video image are weighted summed as the attention features corresponding to the video image, thereby guiding the target segmentation of the video image.

[0059] Obviously, when the above scheme is adopted, with the increasing number of support images in the support set and video images in the query video, the computational complexity and memory consumption in the target segmentation process will increase exponentially, seriously affecting the speed of target segmentation. When the computational complexity and memory consumption are too large, the target segmentation may even fail to be completed.

[0060] By adopting the solution provided in the embodiment of the present application, Figure 2As shown, by introducing a domain agent 13 (selected from the video image), the domain agent 13 is used to fully connect the support images in the support set 11 and the video images in the query video 12, respectively, and the original many-to-many attention is decomposed into two many-to-one attentions. When the number of support images and video images increases, the amount of calculation and memory consumption in the target segmentation process will increase linearly, which helps to reduce the amount of calculation, thereby increasing the speed of target segmentation.

[0061] The video object segmentation method provided in the embodiment of the present application can be used as a preprocessing step for tasks such as video understanding, video content modification, and video retrieval.

[0062] When applied to video understanding scenarios, after segmenting the target object from the video image of the video using the support set containing the target object as a guide, the motion of the target object can be determined based on the change in the position of the target object in the video image; or, the action performed or the state of the target object can be determined based on the morphological changes of the target object in different video images.

[0063] When applied to video content modification scenarios, the support set containing the object to be replaced is used as a guide. After the object to be replaced is segmented from the video image of the video, the object to be replaced can be replaced with a replacement material based on the position of the object to be replaced in the video image, or the object to be replaced can be removed from the video image.

[0064] When applied to video retrieval scenarios, the support set containing the object to be retrieved is used as a guide to segment the object to be retrieved from the video image of the video to be retrieved. The appearance time of the object to be retrieved can be determined based on the timestamp of the video image containing the object to be retrieved. For example, the appearance time of a person, animal or vehicle in the video screen can be determined.

[0065] Of course, in addition to the above application scenarios, the video target segmentation method provided in the embodiment of the present application can also be applied to other video target segmentation scenarios based on a small number of samples, and the embodiment of the present application does not limit the specific application scenarios.

[0066] Figure 3 A schematic diagram of an implementation environment provided by an exemplary embodiment of the present application is shown. The implementation environment includes a terminal 310 and a server 320. The terminal 310 and the server 320 perform data communication via a communication network. Optionally, the communication network may be a wired network or a wireless network, and the communication network may be at least one of a local area network, a metropolitan area network, and a wide area network.

[0067] The terminal 310 is an electronic device that has a video object segmentation requirement, and the electronic device may be a smart phone, a tablet computer, or a personal computer, etc., which is not limited in this embodiment. Figure 3 In the description, the terminal 310 is taken as an example of a personal computer.

[0068] In some embodiments, an application with a video object segmentation function is installed in the terminal 310. When a specific object needs to be segmented from a video image, the user inputs a support set (the support image in the support set contains the specific object and its annotation information) and a query video into the application, and the terminal 310 uploads the support set and the query video to the server 320, which performs video object segmentation and feeds back the segmentation result.

[0069] The server 320 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. Optionally, multiple servers 320 may form a blockchain, and the servers 320 are nodes on the blockchain.

[0070] In some embodiments, the server 320 is used to provide a video target segmentation service for an application installed in the terminal 310. Optionally, the server 320 implements video target segmentation through an encoder 321, an attention model 322, and a decoder 323. The encoder 321 extracts features from the support image in the support set and the video image in the query video, and then inputs the extracted image features into the attention model 322. The attention model 322 determines the attention features of the support set acting on the query video based on the extracted image features. The decoder 323 further performs feature restoration based on the attention features, and finally outputs a segmentation mask 324 corresponding to the video image in the query video (indicating the position of the target object in the video image). The server 320 can feed back the segmentation mask 324 to the terminal 310 so that the terminal 310 displays the segmentation mask 324, or further processes the query video based on the segmentation mask 324.

[0071] Of course, in other possible implementations, the terminal 310 can implement video target segmentation locally (i.e., the encoder 321, the attention model 322, and the decoder 323 are deployed on the terminal side) without the need for the server 320, which is not limited in this embodiment. For the convenience of description, the following embodiments are described by taking the video target segmentation method executed by a computer device as an example.

[0072] Figure 4A flowchart of a video object segmentation method provided by an exemplary embodiment of the present application is shown. This embodiment is described by taking the method used in a computer device as an example, and the method includes the following steps.

[0073] Step 401 , extracting features from a support image in a support set to obtain target object features of a target object in the support image.

[0074] A support set is a set of images consisting of at least one support image, each of which contains a target object. In addition, each support image corresponds to a respective annotation mask, which is used to indicate the location of the target object in the support image. It should be noted that the target object indicates an object of a specific type, such as a person, a car, a cat, a dog, etc.

[0075] In the embodiment of the present application, during the video target segmentation process, the support image is used to guide the segmentation of objects of the same type as the target object from the video image. Among them, the number of support images in the support set will affect the accuracy of the target segmentation. Optionally, the more support images in the support set, the higher the accuracy of the target segmentation.

[0076] In order to prevent objects other than the target object in the support image from interfering with the target segmentation, the computer device extracts the target object features of the target object from the support image. In a possible implementation, for each support image in the support set, the computer device performs feature extraction on the support image to obtain support image features, and further extracts the target object features of the target object from the support image features.

[0077] Illustratively, when it is necessary to segment a "person" in a video image, the support image set is an image containing a "person", and the position of the "person" in the support image is indicated by an annotation mask. The computer device extracts features from each support image in the support set to obtain target object features of the "person".

[0078] Step 402: extract features from the video image in the query video to obtain video image features of the video image.

[0079] The query video is a set of images consisting of at least one continuous video frame, which may or may not contain the target object. The purpose of video object segmentation is to segment the target object from the video image.

[0080] Different from extracting features from the supporting image, the computer device extracts features from each frame of the video image to obtain video image features of the video image, which include features of each object in the video image.

[0081] It should be noted that the above steps 401 and 402 can be executed simultaneously or one after another, and the embodiment of the present application does not limit the execution sequence of the two.

[0082] Step 403, based on the target object features and the video image features, determine a first attention matrix and a second attention matrix, the first attention matrix is ​​the attention matrix between the support set and the domain agent, the second attention matrix is ​​the attention matrix between the query video and the domain agent, and the domain agent is a frame of video image in the query video.

[0083] When a many-to-many attention mechanism is used to determine the attention of the support set on the query video, the computer device needs to calculate a fully connected attention matrix based on the target object features of each support image and the video image features of each video image, and then calculate the attention features based on the fully connected attention matrix.

[0084] However, as the number of supported images and video images continues to increase, the amount of computation required to calculate the fully connected attention matrix will increase exponentially, and accordingly, the memory consumption required to store the fully connected attention matrix will also increase exponentially.

[0085] However, in the embodiment of the present application, the computer device does not directly determine the fully connected attention matrix based on the target object features and the video image features, but introduces a domain agent and uses the domain agent as a bridge to obtain two smaller attention matrices, and uses these two attention matrices to replace the original fully connected attention matrix for subsequent calculation of attention features.

[0086] Optionally, the computer device uses a frame of video image in the query video as a domain agent of the video domain, and then determines a first attention matrix between the support set and the domain agent (indicating the attention of the support set on the domain agent), and a second attention matrix between the query video and the domain agent (indicating the attention of the query video on the domain agent) based on the target object features and the video image features.

[0087] In a possible implementation, the domain proxy is any frame of video image or a specific video image in the query video. For example, the domain proxy is the middle video image in the query video, or the domain proxy is the video image with the richest content in the query video.

[0088] In other possible implementations, the computer device may also generate a domain proxy (image feature) based on the video image features of each frame of the video image to improve the expression quality of the domain proxy for the video domain features, which is not limited in this embodiment.

[0089] Step 404: determine a target attention feature based on the first attention matrix and the second attention matrix, where the target attention feature is used to represent the attention of the support set on the query video.

[0090] The computer device determines the attention of the support set on the query video based on the first attention matrix and the second attention matrix to obtain the target attention feature.

[0091] Since the first attention matrix and the second attention matrix are essentially many-to-one fully connected matrices, the amount of data is smaller than that of the original many-to-many fully connected matrix, which helps to reduce memory consumption when storing matrices. Correspondingly, due to the reduction in computational complexity, the speed of determining the target attention features based on the first attention matrix and the second attention matrix is ​​better than the speed of determining the target attention features based on the original fully connected attention matrix.

[0092] Step 405, target segmentation is performed based on the target attention feature to obtain a target segmentation result, and the target segmentation result is used to characterize the position of the target object in each frame of the video image.

[0093] Furthermore, the computer device performs target segmentation on each frame of the video image based on the target attention feature to obtain a target segmentation result indicating the location of the target object. When the video image contains the target object, the target segmentation result corresponding to the video image contains the location information of the target object; when the video image does not contain the target object, the target segmentation result corresponding to the video image is empty.

[0094] Illustratively, when the target object contained in the supporting image is a "person", the computer device performs target segmentation based on the target attention feature to determine the position of the "person" in each frame of the video image.

[0095] In an embodiment of the present application, feature extraction is performed on the support images in the support set and the video images in the query video respectively. After obtaining the target object features and the video image features, a frame of video images in the query video is used as a domain agent. Based on the target object features and the video image features, a first attention matrix between the support set and the domain agent and a second attention matrix between the query video and the domain agent are determined. Then, based on the first attention matrix and the second attention matrix, the target attention features of the support set acting on the query video are determined. Finally, the target attention features are used to perform target segmentation to obtain the position of the target object in each frame of the video image. By introducing the domain agent, the fully connected attention matrix between the support set and the query video under the many-to-many attention mechanism is decomposed into two many-to-one attention matrices bridged by the domain agent. Therefore, as the number of support images and video images increases, the growth mode of computational complexity and memory consumption in the target segmentation process changes from exponential growth to linear growth, thereby reducing the probability of being unable to perform target segmentation due to excessive computational complexity, thereby achieving the effect of increasing the target segmentation speed.

[0096] In one possible implementation, the feature extraction process is performed by an encoder, the target attention feature determination process is performed by an attention model (including a first convolutional layer and a second convolutional layer), and the target segmentation process is performed by a decoder. The video target segmentation process is described in detail below using an exemplary embodiment.

[0097] Figure 5 A flowchart of a video object segmentation method provided by another exemplary embodiment of the present application is shown. This embodiment is described by taking the method used in a computer device as an example, and the method includes the following steps.

[0098] Step 501: extract features of support images in a support set through an encoder to obtain support image features of the support images.

[0099] Optionally, the encoder is a neural network composed of multiple convolutional layers. Through the convolution kernel, the image or feature map is processed by convolution pooling activation, etc., and the low-level image features are gradually extracted to the high-level image features. As the feature extraction continues to deepen, the extracted image features become more and more abstract (that is, the high-level image features are more abstract than the low-level image features), and finally the supporting image features are obtained.

[0100] In a possible implementation, the encoder is based on ResNet-50 and is used to extract common features of an image. In other possible implementations, the encoder may also adopt other network structures, and the embodiment of the present application does not limit the specific structure of the encoder.

[0101] Indicative, such as Figure 6As shown, the computer device inputs the supporting images 601 (3 images in total) in the supporting set into the encoder 602 , and obtains the supporting image features 603 output by the encoder 602 .

[0102] Step 502 : obtaining target object features of the target object based on the support image features and the annotation mask corresponding to the support image. The annotation mask is used to indicate the position of the target object in the support image.

[0103] In order to prevent other objects other than the target object in the support image from interfering with the subsequent target segmentation, the computer device further extracts the target object features from the support image features. In this embodiment, each support image corresponds to its own annotation mask, and the computer device uses the annotation mask to extract the target object features from the support image features.

[0104] The annotation mask may be a binary mask, and the pixel value in the annotation mask is used to indicate whether the corresponding pixel is the target object. For example, the pixel in the 0-value area of ​​the binary mask does not belong to the target object, while the pixel in the 1-value area belongs to the target object.

[0105] In a possible implementation, the computer device performs element-wise multiplication of the supporting image features and the corresponding annotation masks to obtain the target object features.

[0106] Indicative, such as Figure 6 As shown, the position of the target object "cow" is marked in the annotation mask 604 of the supporting image 601. After extracting the feature of the supporting image 601 to obtain the supporting image feature 603, the computer device multiplies the supporting image feature 603 by the annotation mask at the element level to obtain the target object feature.

[0107] Step 503: extract features of the video image in the query video through an encoder to obtain video image features of the video image.

[0108] Similar to the process of extracting features of the supported image, the computer device extracts features of the video image in the query video through an encoder to obtain video image features corresponding to each frame of the video image. The computer device may be an encoder, that is, the supported image and the video image use the same encoder for feature extraction.

[0109] Indicative, such as Figure 6 As shown, the computer device inputs the video image 605 (a total of 3 consecutive frames) in the query video into the encoder 602 to obtain the video image feature 606 output by the encoder 602.

[0110] In the embodiment of the present application, the attention model used for extracting attention features is called a domain agent attention model (DAA), which includes two branches, namely, a first convolution layer for processing target object features and a second convolution layer for processing video image features. After completing feature extraction, the computer device inputs the target object features into the first convolution layer and the video image features into the second convolution layer.

[0111] Optionally, the first convolutional layer and the second convolutional layer have the same structure but different network parameters. For example, the first convolutional layer and the second convolutional layer may be DenseNets with the same structure but different parameters. The embodiment of the present application does not limit the specific structure of the first convolutional layer and the second convolutional layer.

[0112] Step 504: Map the target object feature to a first key-value pair feature through a first convolutional layer, where the first key-value pair feature is composed of a first key feature and a first value feature.

[0113] The first convolutional layer performs feature mapping on the input target object features to obtain the first key-value pair (KV) features corresponding to the target object features. Each set of first key-value pair features contains the first key feature and the first value feature. The key feature is used to calculate the attention weight with the query feature (query, q), and the multiplication of the value feature and the attention weight indicates that attention is given to the image.

[0114] Indicative, such as Figure 6 As shown, the first convolution layer 607 maps the input target object features to obtain a first key feature 608 and a first value feature 609.

[0115] Step 505 , mapping the video image features into query features and second key-value pair features through a second convolutional layer, where the second key-value pair features are composed of a second key feature and a second value feature.

[0116] When the video image features are mapped through the second convolutional layer, in addition to the second key-value pair features, the query features are also obtained.

[0117] Indicative, such as Figure 6 As shown, the second convolutional layer 610 maps the input target object features to obtain query features 611, second key features 612 and second value features 613.

[0118] In one example, when the support set contains M supporting images, and the query video When N frames of video images are included, the target object features and video image features are mapped respectively, and the key-value pair features and query features obtained are as follows:

[0119]

[0120] in, is the height, is the width, is the number of channels of the query feature, is the number of channels of the key feature, is the number of channels of the value feature.

[0121] Step 506, generating a first attention matrix based on the first key-value pair features and the query features.

[0122] Under the many-to-many attention mechanism, the attention features The calculation process is as follows:

[0123]

[0124] in, To query features, is the key feature, is the value feature, is the number of channels of the key feature, is the softmax function, This is the fully connected attention matrix.

[0125] In the embodiment of the present application, the computer device uses two attention matrices (connected by a domain proxy) to replace the fully connected attention matrix. The replacement matrix for improving the fully connected attention matrix can be expressed as:

[0126]

[0127] in, To query features With Domain Agent The attention matrix between For the field agent Key Features The attention matrix between .

[0128] Accordingly, the computer device determines a first attention matrix based on the query feature corresponding to the domain agent and the first key feature corresponding to the supporting image. In a possible implementation, this step may include the following steps:

[0129] 1. Perform feature sampling on the query features to obtain the proxy query features corresponding to the domain proxy.

[0130] In a possible implementation, the computer device selects a frame of video image from the query video as a domain proxy, and performs feature sampling on the query feature to obtain a proxy query feature corresponding to the domain proxy.

[0131] Indicative, such as Figure 6 As shown, the computer device performs feature sampling on the query feature 611 to obtain the agent query feature 614 corresponding to the domain agent.

[0132] 2. Generate the first attention matrix based on the dot product result of the proxy query feature and the first key feature.

[0133] In a possible implementation, the computer device performs a scale dot-product on the proxy query feature and each first key feature, and performs a softmax process on the dot-product result to obtain a first attention matrix. The formula for generating the first attention matrix is ​​as follows:

[0134]

[0135] in, is the first attention matrix, For the field agent The corresponding proxy query features, is the first key feature, is the number of channels of the first key feature, is the softmax function.

[0136] Indicative, such as Figure 6 As shown, the computer device performs a dot product operation on the proxy query feature 614 and the first key feature 608, thereby generating a first attention matrix based on the dot product operation result. .

[0137] Step 507, generating a second attention matrix based on the query feature and the second key-value pair feature.

[0138] The first attention matrix is ​​used to characterize the degree of influence of the support set on the attention of the domain agent. Since the domain agent is only a frame in the query video, the computer device needs to further determine the degree of influence of the query video on the attention of the domain agent, thereby obtaining the second attention matrix.

[0139] In a possible implementation, the computer device determines the second attention matrix based on the query feature corresponding to the query video and the second key feature corresponding to the domain agent. Optionally, this step may include the following steps:

[0140] 1. Perform feature sampling on the second key feature to obtain the proxy key feature corresponding to the domain proxy.

[0141] In a possible implementation, the computer device selects a frame of video image from the query video as a domain proxy, and performs feature sampling on the second key feature to obtain a proxy key feature corresponding to the domain proxy.

[0142] Indicative, such as Figure 6 As shown, the computer device performs feature sampling on the second key feature 612 to obtain a proxy key feature 615 corresponding to the domain proxy.

[0143] 2. Generate the second attention matrix based on the dot product result of the surrogate key feature and the query feature.

[0144] In one possible implementation, the computer device performs a scale dot-product on the proxy key feature and each query feature, and performs a softmax process on the dot-product result to obtain a second attention matrix. The formula for generating the second attention matrix is ​​as follows:

[0145]

[0146] in, is the second attention matrix, For the field agent The corresponding surrogate key features, To query features, is the number of channels of the surrogate key feature, is the softmax function.

[0147] Indicative, such as Figure 6 As shown, the computer device performs a dot product operation on the surrogate key feature 615 and the query feature 611, thereby generating a second attention matrix based on the dot product operation result. .

[0148] Step 508, obtaining the target attention feature based on the matrix product result of the first attention matrix, the second attention matrix and the first value feature.

[0149] Since the first attention matrix and the second attention matrix are used instead of the original fully connected attention matrix, the formula for determining the target attention feature can be transformed as follows:

[0150]

[0151] in, is the first value feature corresponding to the support set.

[0152] Accordingly, in the process of calculating the target attention feature, the computer device first calculates Then with Performing matrix multiplication can avoid calculating and storing the fully connected attention matrix.

[0153] From the perspective of computational complexity, when the support set contains M supporting images, and the query video When N frames of video images are included, the computational complexity is given by Reduced to .

[0154] Indicative, such as Figure 6 As shown, the computer device first combines the first value feature 609 with the first attention matrix Matrix multiplication (MatMul), and then further combine the result with the second attention matrix Matrix multiplication to obtain the target attention feature .

[0155] Step 509, perform feature concatenation on the target attention feature and the second value feature to obtain the target feature of the query video.

[0156] Furthermore, the computer device performs feature concatenation (concat) on the target attention feature and the second value feature of the query video to obtain the target feature of the query video.

[0157] Indicative, such as Figure 6 As shown, the computer device processes the second value feature 613 and the target attention feature The splicing is performed as the input of the decoder 616 during the subsequent target segmentation.

[0158] Step 510, upsampling and convolution processing are performed on the target features through the decoder to obtain a segmentation mask corresponding to each frame of the video image. The segmentation mask is a binary mask, and the pixel value in the segmentation mask is used to represent whether the corresponding pixel point belongs to the target object.

[0159] In this embodiment, during the target segmentation process, the decoder gradually performs upsampling and convolution processing on the target features, thereby restoring the target features to the original size of the video image, and finally outputs a segmentation mask of the video image. The segmentation mask is similar to the annotation mask corresponding to the support image, that is, the position of the target object in the video image is indicated by the pixel value.

[0160] Optionally, the decoder may adopt a decoder structure in FCN (fully connected network) or SegNet (semantic segmentation network). The embodiments of the present application do not limit the specific network structure of the decoder.

[0161] Since the target features finally input to the decoder are too abstract (high-level features), which is not conducive to subsequent feature decoding, in an embodiment of the present application, a skip-connection is used between the encoder and the decoder for feature extraction. During the feature decoding process, the low-level image features extracted by the middle layer of the encoder are passed to the decoder, thereby incorporating more low-level image semantic information into the feature decoding process, improving the feature decoding quality, and further improving the accuracy of the target segmentation results.

[0162] Indicative, such as Figure 6 As shown, there is a jump connection between the encoder 602 and the decoder 616. When the decoder 616 performs target segmentation, the encoder 602 inputs the low-level image semantic information (feature map) of the query video into the network layer corresponding to the decoder 616, and the decoder 616 performs target segmentation based on the low-level image semantic information and target features, and finally outputs a segmentation mask 617. The white area in the segmentation mask 617 is the segmented "cow".

[0163] In this embodiment, the computer device maps the target object features and the video image features into query features and key-value pair features respectively, and then uses the query features and the key-value pair features to generate a first attention matrix and a second attention matrix bridged by a domain agent, and then uses the first attention matrix and the second attention matrix to calculate the attention features, thereby reducing the memory space occupied by storing the attention matrix, reducing the computational complexity of the attention feature calculation process, and improving the speed of target segmentation.

[0164] Furthermore, in this embodiment, by jump-connecting the encoder and the decoder, the decoder can incorporate more low-level image semantic features during the object segmentation process, which helps to improve the accuracy of object segmentation.

[0165] The above embodiment describes the application process of video object segmentation. The following uses an exemplary embodiment to describe the training process before application.

[0166] Figure 7 A flowchart of a video object segmentation method provided by another exemplary embodiment of the present application is shown. This embodiment is described by taking the method used in a computer device as an example, and the method includes the following steps.

[0167] Step 701: extract features from a first sample support image in a first sample support set to obtain first sample object features of a first sample object in the first sample support image.

[0168] In a possible implementation, the computer device extracts features from a first sample support image in a first sample support set (ie, a training set) through an encoder to obtain first sample object features.

[0169] The process of extracting the first sample object feature from the first sample supporting image may refer to the above-mentioned embodiment of the application side, and this embodiment will not be described in detail here.

[0170] Step 702: extract features from the sample video image in the sample query video to obtain sample video image features of the sample video image.

[0171] In a possible implementation, the computer device extracts features of the sample video images in the sample query support set through an encoder to obtain features of the sample video images.

[0172] The process of extracting sample video image features from sample video images may refer to the above-mentioned embodiment of the application side, and this embodiment will not be described in detail here.

[0173] Step 703: Input the first sample object feature and the sample video image feature into the attention model to obtain the sample attention feature, which is used to represent the attention of the first sample support set on the sample query video. The attention model is used to determine the first sample attention matrix and the second sample attention matrix based on the first sample target object feature and the sample video image feature, and determine the sample attention feature based on the first sample attention matrix and the second sample attention matrix. The first sample attention matrix is ​​the attention matrix between the first sample support set and the sample domain agent, and the second sample attention matrix is ​​the attention matrix between the sample query video and the sample domain agent. The sample domain agent is a frame of sample video image in the sample query video.

[0174] In one possible implementation, the attention model first determines a first sample attention matrix between a first sample support set and a sample domain agent, and a second sample attention matrix between a sample query video and a sample domain agent based on first sample object features and sample video image features, and then determines sample attention features based on the first sample attention matrix and the second sample attention matrix.

[0175] Among them, the process of using the attention model to generate attention features can refer to the above-mentioned application side embodiment, and this embodiment will not be repeated here.

[0176] Step 704, based on the sample attention feature, target segmentation is performed through a decoder to obtain a sample segmentation result, and the sample segmentation result is used to characterize the position of the first sample object in each frame of the sample video image.

[0177] The process of using the decoder to segment the target may refer to the above-mentioned application side embodiment, and this embodiment will not be repeated here.

[0178] Optionally, the sample segmentation result is a sample segmentation mask, and the sample annotation mask is used to indicate the position of the sample object in the sample video image.

[0179] Step 705: train the attention model and decoder based on the sample segmentation results and the sample annotation information corresponding to the sample video image.

[0180] In this embodiment, the computer device uses the sample annotation information corresponding to the sample video image as supervision, determines the loss based on the sample segmentation result and the sample annotation information, and trains the attention model and the decoder based on the loss.

[0181] In one possible implementation, the sample annotation information is a sample annotation mask (used to characterize the position of the sample object in the sample video image), and the sample segmentation result is a sample segmentation mask. Accordingly, the computer device determines the loss based on the sample annotation mask and the sample segmentation mask, and trains the attention model and the decoder through a back-propagation algorithm.

[0182] To summarize, in the embodiments of the present application, feature extraction is performed on the support images in the support set and the video images in the query video respectively. After obtaining the target object features and the video image features, a frame of video images in the query video is used as a domain agent. Based on the target object features and the video image features, a first attention matrix between the support set and the domain agent and a second attention matrix between the query video and the domain agent are determined. Then, based on the first attention matrix and the second attention matrix, the target attention features of the support set acting on the query video are determined. Finally, the target attention features are used to perform target segmentation to obtain the position of the target object in each frame of the video image. By introducing the domain agent, the fully connected attention matrix between the support set and the query video under the many-to-many attention mechanism is decomposed into two many-to-one attention matrices bridged by the domain agent. Therefore, as the number of support images and video images increases, the growth mode of computational complexity and memory consumption in the target segmentation process changes from exponential growth to linear growth, thereby reducing the probability of being unable to perform target segmentation due to excessive computational complexity, thereby achieving the effect of increasing the target segmentation speed.

[0183] In a possible implementation, the attention model includes a first convolutional layer and a second convolutional layer, and the computer device may obtain the sample attention feature through the attention model by including the following steps:

[0184] 1. Mapping the first sample object feature to a first sample key-value pair feature through a first convolutional layer, wherein the first sample key-value pair feature is composed of a first sample key feature and a first sample value feature.

[0185] The implementation of this step may refer to the above step 504, and this embodiment will not be described in detail here.

[0186] 2. Mapping the sample video image features into sample query features and second sample key-value pair features through the second convolutional layer, where the second sample key-value pair features are composed of the second sample key features and the second sample value features.

[0187] The implementation of this step may refer to the above step 505, and this embodiment will not be described in detail here.

[0188] 3. Generate a first sample attention matrix based on the first sample key-value pair features and sample query features.

[0189] In one possible implementation, the computer device performs feature sampling on the sample query feature to obtain the sample proxy query feature corresponding to the sample domain proxy, thereby generating a first sample attention matrix based on the dot product result of the sample proxy query feature and the first sample key feature.

[0190] Among them, the detailed process of generating the first sample attention matrix can be referred to step 506, and this embodiment will not be repeated here.

[0191] 4. Generate the second sample attention matrix based on the sample query features and the second sample key-value pair features.

[0192] In one possible implementation, the computer device performs feature sampling on the second sample key feature to obtain a sample proxy key feature corresponding to the sample domain proxy, thereby generating a second sample attention matrix based on a dot product result of the sample proxy key feature and the sample query feature.

[0193] Among them, the detailed process of generating the second sample attention matrix can be referred to step 507, and this embodiment will not be repeated here.

[0194] 5. Determine the sample attention features based on the first sample attention matrix and the second sample attention matrix.

[0195] In a possible implementation, the computer device obtains the sample attention feature based on the matrix product result of the first sample attention matrix, the second sample attention matrix and the first sample value feature. The detailed implementation can refer to step 508, which will not be described in detail in this embodiment.

[0196] Accordingly, target segmentation by the decoder may include the following steps:

[0197] 1. Concatenate the sample attention feature and the second sample value feature to obtain the sample feature of the sample query video.

[0198] Optionally, the computer device performs feature concatenation (concat) on the sample attention feature and the second sample value feature of the sample query video to obtain the sample feature of the sample query video.

[0199] 2. The sample features are upsampled and convolved through the decoder to obtain a sample segmentation mask corresponding to each frame of the sample video image. The sample segmentation mask is a binary mask, and the pixel value in the sample segmentation mask is used to indicate whether the corresponding pixel point belongs to the first sample object.

[0200] In this embodiment, the decoder gradually performs upsampling and convolution processing on the sample features during the target segmentation process, thereby restoring the sample features to the original size of the sample video image, and finally outputs a sample segmentation mask of the sample video image.

[0201] Since the sample features finally input to the decoder are too abstract (high-level features), which is not conducive to subsequent feature decoding, in the embodiment of the present application, a skip connection is used between the encoder and the decoder for feature extraction. During the feature decoding process, the low-level image features extracted by the encoder intermediate layer are passed to the decoder, thereby incorporating more low-level image semantic information into the feature decoding process, improving the quality of feature decoding, and further improving the accuracy of the target segmentation result. The detailed implementation method can refer to the above steps 509 to 510, and this embodiment will not be repeated here.

[0202] Regarding the loss function used in the training process, in one possible implementation, when the sample segmentation result is a sample segmentation mask and the sample annotation information is a sample annotation mask, the computer device determines the segmentation loss based on the cross entropy (CE) loss and the intersection over Union (IoU) loss between the sample segmentation mask and the sample annotation mask, thereby training the attention model and the decoder based on the segmentation loss.

[0203] Schematically, the cross entropy loss between the sample segmentation mask and the sample annotation mask can be expressed as:

[0204]

[0205] The intersection-over-union loss between the sample segmentation mask and the sample annotation mask can be expressed as:

[0206]

[0207] in, is the sample segmentation mask, Label the samples with masks.

[0208] Optionally, the loss weights corresponding to the cross entropy loss and the intersection-over-union ratio loss in the segmentation loss are different, for example, the loss weight of the cross entropy loss is 5, while the loss weight of the intersection-over-union ratio loss is 1. The embodiment of the present application does not limit the specific loss weights.

[0209] Furthermore, in order to further improve the training quality and thus improve the accuracy of target segmentation during the application process, in a possible implementation, the computer device uses a meta-learning + online-learning method to perform model training and testing. Figure 8 As shown, the testing process after the training process may include the following steps.

[0210] Step 801: extract features from a second sample support image in a second sample support set by using an encoder to obtain second sample object features of a second sample object in the sample support image.

[0211] The second support sample set is different from the first support sample set, that is, the object type of the sample objects in the first support sample set is different from the object type of the sample objects in the second support sample set.

[0212] The process of the computer device extracting features from the second sample support image through the encoder may refer to the above step 701, which will not be described in detail in this embodiment.

[0213] Step 802: Select a sampling support image from the second sample support image, and perform feature extraction on the sampling support image through an encoder to obtain sampling image features of the sampling support image.

[0214] During the online learning process, the computer samples the second sample support image to obtain a sample support image, and uses the sample support image as a sample video image, and then uses an encoder to extract features from the sample support image to obtain corresponding sample image features.

[0215] The process of the computer device extracting features from the sampled support image through the encoder may refer to the above step 702, which will not be described in detail in this embodiment.

[0216] Step 803: Input the second sample object features and the sampled image features into the attention model to obtain the sampled attention features, where the sampled attention features are used to represent the attention of the second sample support set on the sampled support image.

[0217] Further, the computer device uses the attention model trained in the training process to perform feature mapping on the second sample object features and the sampled image features, and calculates the sampled attention features based on the mapped Q, K, and V, wherein the sampled support image is the sample domain proxy. The process of calculating the sampled attention features can refer to the above embodiment, and this embodiment will not be repeated here.

[0218] Step 804, based on the sampled attention features, target segmentation is performed through a decoder to obtain a sampled segmentation result, and the sampled segmentation result is used to characterize the position of the second sample object in the sampled support image.

[0219] Similar to the training process, the computer device decodes the sampled attention features through a decoder to obtain the sampled segmentation results corresponding to the sampled support image. The specific implementation process can be referred to the above embodiment and will not be repeated here.

[0220] Step 805 : training an encoder based on the sampled segmentation result and the support image annotation information corresponding to the sampled support image.

[0221] Different from the training of the attention model and decoder during the training process, during the testing process, the computer equipment trains the encoder based on the sampling segmentation results and supporting image annotation information to improve the encoder's feature extraction ability for new types of objects, that is, to improve the encoder's feature extraction quality.

[0222] Indicative, such as Fig. 9 As shown, during the meta-learning process (i.e., the training process), the computer device inputs the first sample support set 91 and the sample query video 92 into the encoder 93 respectively, and the encoder 93 inputs the feature extraction result into the attention model 94, and the attention model 94 calculates the attention feature, and then the attention feature is input into the decoder 95, and the decoder 95 performs target segmentation to obtain the sample segmentation mask 96. The computer device fixes the parameters of the encoder 93 and only adjusts the parameters of the attention model 94 and the decoder 95.

[0223] During the online learning process (i.e., the testing process), the computer device samples the second sample support set 97 to obtain a sample support image 98, and inputs the second sample support set 97 and the sample support image 98 into the encoder 93 respectively. The encoder 93 inputs the feature extraction result into the attention model 94, and the attention model 94 calculates the attention feature, thereby inputting the attention feature into the decoder 95, and the decoder 95 performs target segmentation to obtain a sample segmentation mask 96. The computer device fixes the parameters of the attention model 94 and the decoder 95, and only adjusts the parameters of the encoder 93.

[0224] In one possible implementation, the computer device is trained based on the Youtube-VIS dataset. The dataset includes 2238 videos, 3774 instances, and covers 40 object categories. The dataset is evenly divided into four groups and cross-validated on all groups. Each round of validation contains 30 training categories and 10 test categories. During the training process, similarity (J) and contour accuracy (F) are used to measure the segmentation performance. During the training process, k-shot means using k support images for segmentation, and in some embodiments, the default setting is 5-shot.

[0225] In addition, the computer device uses Adam as the optimizer for model training, and the learning rate in the meta-learning stage is set to 1e-5, and the learning rate in the online learning stage is set to 5e-6. The batch size of meta-learning is 4, and the number of training iterations is 75,000; the batch size of online learning is 1, and the number of training iterations is 100. The image size of the model input is (241, 425).

[0226] The memory consumption and computing time experiments were conducted on the model trained in the above way. The experimental results are as follows: Fig.10 and Fig.11 shown.

[0227] from Fig.10 It can be seen that when the number of supported images and video images is 40, the memory consumption under the solution of the present application is only 8.79 GB, while the related technical solution has used 10.76 GB when the number of images is 20. After visualizing the growth trend of the above two solutions, it can be found that the memory consumption under the related technical solution increases exponentially, while the memory consumption under the solution of the present application increases linearly, which supports the input of more images under the same memory condition and provides the possibility of parallel computing.

[0228] from Fig.11 It can be seen that under different k-shot settings, tests are performed on query videos with different numbers of frames (5 to 40 frames). Both the related technology and the present application solution can benefit from processing more video images, and the present application solution takes less time than the related technology solution.

[0229] From the perspective of segmentation quality, a comparison of the related art and the solution provided by this application in terms of the two indicators J and F is shown in Table 1.

[0230] Table 1

[0231]

[0232] As can be seen from Table 1, the present application scheme improves the contour accuracy by 1.4% and the regional similarity by 1.1% compared with the related technical scheme.

[0233] In addition, in order to verify the improvement of model training quality by online learning, experiments were conducted on two cases: adding online learning and not adding online learning. The experimental results are shown in Table 2.

[0234] Table 2

[0235]

[0236] It can be clearly seen from Table 2 that after adding online learning, the accuracy of target segmentation is significantly improved.

[0237] Finally, different schemes are used to segment the target of the same video, and the target segmentation results are as follows: Fig.12 As shown. It can be seen that when the image contains a single object (i.e., the elephant in the picture), the segmentation results of PFENet, PPNet, and FAN are incomplete. Although PMMs can complete the segmentation, the segmentation boundary cannot be accurately aligned with the object. Under the present application scheme, better results are obtained in terms of completeness and edge consistency. When the image contains two objects (i.e., the person and the vehicle in the picture), the segmentation results of PMMS, PPNet, and FAN mistakenly include part of the person's area, while the present application scheme can accurately segment the vehicle.

[0238] Fig.13 : is a structural block diagram of a video object segmentation device provided by an exemplary embodiment of the present application, the device comprising:

[0239] A first extraction module 1301 is used to extract features from a support image in a support set to obtain target object features of a target object in the support image;

[0240] The second extraction module 1302 is used to extract features of the video image in the query video to obtain video image features of the video image;

[0241] A first matrix determination module 1303 is used to determine a first attention matrix and a second attention matrix based on the target object feature and the video image feature, wherein the first attention matrix is ​​an attention matrix between the support set and the domain agent, and the second attention matrix is ​​an attention matrix between the query video and the domain agent, and the domain agent is a frame of video image in the query video;

[0242] A first feature determination module 1304 is used to determine a target attention feature based on the first attention matrix and the second attention matrix, wherein the target attention feature is used to represent the attention of the support set on the query video;

[0243] The first segmentation module 1305 is used to perform target segmentation based on the target attention feature to obtain a target segmentation result, and the target segmentation result is used to characterize the position of the target object in the video image of each frame.

[0244] Optionally, the first matrix determination module 1303 includes:

[0245] A first mapping unit, configured to map the target object feature into a first key-value pair feature through a first convolutional layer, wherein the first key-value pair feature is composed of a first key feature and a first value feature;

[0246] A second mapping unit, configured to map the video image feature into a query feature and a second key-value pair feature through a second convolutional layer, wherein the second key-value pair feature is composed of a second key feature and a second value feature;

[0247] A first generating unit, configured to generate the first attention matrix based on the first key-value pair feature and the query feature;

[0248] The second generating unit is used to generate the second attention matrix based on the query feature and the second key-value pair feature.

[0249] Optionally, the first generating unit is used to:

[0250] Performing feature sampling on the query feature to obtain the agent query feature corresponding to the domain agent;

[0251] The first attention matrix is ​​generated based on a dot product result of the proxy query feature and the first key feature.

[0252] Optionally, the second generating unit is used to:

[0253] Performing feature sampling on the second key feature to obtain a proxy key feature corresponding to the domain proxy;

[0254] The second attention matrix is ​​generated based on the dot product result of the surrogate key feature and the query feature.

[0255] Optionally, the first feature determination module 1304 is configured to:

[0256] The target attention feature is obtained based on the matrix product result of the first attention matrix, the second attention matrix and the first value feature.

[0257] Optionally, the first segmentation module 1305 is used to:

[0258] Performing feature concatenation on the target attention feature and the second value feature to obtain a target feature of the query video;

[0259] The target features are upsampled and convolved by a decoder to obtain a segmentation mask corresponding to each frame of the video image, wherein the segmentation mask is a binary mask, and the pixel value in the segmentation mask is used to indicate whether the corresponding pixel point belongs to the target object.

[0260] Optionally, the video image features are extracted by an encoder, and the encoder and the decoder are jump-connected.

[0261] Optionally, the first extraction module 1301 is used to:

[0262] Extracting features of the support image in the support set by an encoder to obtain support image features of the support image;

[0263] The target object feature of the target object is obtained based on the supporting image feature and a marked mask corresponding to the supporting image, wherein the marked mask is used to indicate the position of the target object in the supporting image.

[0264] Fig.14 : is a structural block diagram of a video object segmentation device provided by an exemplary embodiment of the present application, the device comprising:

[0265] A third extraction module 1401 is used to extract features from a first sample support image in a first sample support set to obtain a first sample object feature of a first sample object in the first sample support image;

[0266] The fourth extraction module 1402 is used to extract features of the sample video image in the sample query video to obtain sample video image features of the sample video image;

[0267] A second feature determination module 1403 is used to input the first sample object feature and the sample video image feature into an attention model to obtain a sample attention feature, wherein the sample attention feature is used to represent the attention of the first sample support set on the sample query video, and the attention model is used to determine a first sample attention matrix and a second sample attention matrix based on the first sample target object feature and the sample video image feature, and determine the sample attention feature based on the first sample attention matrix and the second sample attention matrix, wherein the first sample attention matrix is ​​an attention matrix between the first sample support set and a sample domain agent, and the second sample attention matrix is ​​an attention matrix between the sample query video and the sample domain agent, and the sample domain agent is a frame of sample video image in the sample query video;

[0268] A second segmentation module 1404 is used to perform target segmentation through a decoder based on the sample attention feature to obtain a sample segmentation result, wherein the sample segmentation result is used to characterize the position of the first sample object in each frame of the sample video image;

[0269] The first training module 1405 is used to train the attention model and the decoder based on the sample segmentation result and the sample annotation information corresponding to the sample video image.

[0270] Optionally, the attention model includes a first convolutional layer and a second convolutional layer;

[0271] The second feature determination module 1403 includes:

[0272] a third mapping unit, configured to map the first sample object feature into a first sample key-value pair feature through the first convolutional layer, wherein the first sample key-value pair feature is composed of a first sample key feature and a first sample value feature;

[0273] a fourth mapping unit, configured to map the sample video image feature into a sample query feature and a second sample key-value pair feature through the second convolutional layer, wherein the second sample key-value pair feature is composed of a second sample key feature and a second sample value feature;

[0274] A third generating unit, configured to generate the first sample attention matrix based on the first sample key-value pair feature and the sample query feature;

[0275] a fourth generating unit, configured to generate the second sample attention matrix based on the sample query feature and the second sample key-value pair feature;

[0276] A feature determination unit is used to determine a sample attention feature based on the first sample attention matrix and the second sample attention matrix.

[0277] Optionally, the second segmentation module 1404 is configured to:

[0278] Performing feature concatenation on the sample attention feature and the second sample value feature to obtain a sample feature of the sample query video;

[0279] The sample features are upsampled and convolved through a decoder to obtain a sample segmentation mask corresponding to each frame of the sample video image, wherein the sample segmentation mask is a binary mask, and the pixel value in the sample segmentation mask is used to indicate whether the corresponding pixel point belongs to the first sample object.

[0280] Optionally, the sample segmentation result is a sample segmentation mask, the sample annotation information is a sample annotation mask, and the sample annotation mask is used to indicate the position of the sample object in the sample video image;

[0281] The first training module 1405 is used to:

[0282] Determining a segmentation loss based on a cross entropy loss and an intersection-over-union loss between a sample segmentation mask and the sample annotation mask;

[0283] The attention model and the decoder are trained based on the segmentation loss.

[0284] Optionally, the first sample object feature and the sample video image feature are extracted by an encoder;

[0285] The device also includes a second training module, which is used to:

[0286] During the test, the encoder is used to extract features from a second sample support image in a second sample support set to obtain second sample object features of a second sample object in the sample support image;

[0287] Selecting a sampling support image from the second sample support image, and performing feature extraction on the sampling support image by the encoder to obtain sampling image features of the sampling support image;

[0288] Inputting the second sample object feature and the sampled image feature into the attention model to obtain a sampled attention feature, where the sampled attention feature is used to represent the attention of the second sample support set on the sampled support image;

[0289] Based on the sampling attention feature, performing target segmentation through the decoder to obtain a sampling segmentation result, wherein the sampling segmentation result is used to characterize the position of the second sample object in the sampling support image;

[0290] The encoder is trained based on the sampling segmentation result and the support image annotation information corresponding to the sampled support image.

[0291] It should be noted that: the device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the implementation process thereof is detailed in the method embodiment, which will not be repeated here.

[0292] Please refer to Fig.15, which shows a schematic diagram of the structure of a computer device provided by an exemplary embodiment of the present application. Specifically, the computer device 1500 includes a central processing unit (CPU) 1501, a system memory 1504 including a random access memory 1502 and a read-only memory 1503, and a system bus 1505 connecting the system memory 1504 and the central processing unit 1501. The computer device 1500 also includes a basic input / output system (I / O system) 1506 that helps transmit information between various devices in the computer, and a large-capacity storage device 1507 for storing an operating system 1513, an application program 1514, and other program modules 1515.

[0293] The basic input / output system 1506 includes a display 1508 for displaying information and an input device 1509 such as a mouse and a keyboard for user inputting information. The display 1508 and the input device 1509 are connected to the central processing unit 1501 through an input / output controller 1510 connected to the system bus 1505. The basic input / output system 1506 may also include an input / output controller 1510 for receiving and processing inputs from a plurality of other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 1510 also provides output to a display screen, a printer, or other types of output devices.

[0294] The mass storage device 1507 is connected to the central processing unit 1501 through a mass storage controller (not shown) connected to the system bus 1505. The mass storage device 1507 and its associated computer readable media provide non-volatile storage for the computer device 1500. That is, the mass storage device 1507 may include a computer readable medium (not shown) such as a hard disk or drive.

[0295] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include random access memory (RAM), read-only memory (ROM), flash memory or other solid-state storage technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, tape cassettes, magnetic tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will know that the computer storage medium is not limited to the above. The above-mentioned system memory 1504 and mass storage device 1507 can be collectively referred to as memory.

[0296] The memory stores one or more programs, and the one or more programs are configured to be executed by one or more central processing units 1501. The one or more programs contain instructions for implementing the above-mentioned methods. The central processing unit 1501 executes the one or more programs to implement the methods provided by the above-mentioned various method embodiments.

[0297] According to various embodiments of the present application, the computer device 1500 can also be connected to a remote computer on the network through a network such as the Internet. That is, the computer device 1500 can be connected to the network 1512 through the network interface unit 1511 connected to the system bus 1505, or the network interface unit 1511 can be used to connect to other types of networks or remote computer systems (not shown).

[0298] The memory also includes one or more programs, which are stored in the memory and include steps executed by a computer device in the method provided in the embodiment of the present application.

[0299] An embodiment of the present application further provides a computer-readable storage medium, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement the video object segmentation method described in any of the above embodiments.

[0300] Optionally, the computer readable storage medium may include: ROM, RAM, solid state drives (SSD) or optical disks, etc. Among them, RAM may include resistance random access memory (ReRAM) and dynamic random access memory (DRAM).

[0301] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the video object segmentation method described in the above embodiment.

[0302] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0303] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A video object segmentation method, characterized in that: The method comprises: Performing feature extraction on the support image in the support set to obtain target object features of the target object in the support image; Extracting features of the video image in the query video to obtain video image features of the video image; Mapping the target object feature into a first key-value pair feature through a first convolutional layer, where the first key-value pair feature consists of a first key feature and a first value feature; Mapping the video image feature into a query feature and a second key-value pair feature through a second convolutional layer, where the second key-value pair feature consists of a second key feature and a second value feature; Based on the first key-value pair feature and the query feature, generating a first attention matrix, wherein the first attention matrix is ​​an attention matrix between the support set and the domain agent, and the domain agent is a frame of video image in the query video; Based on the query feature and the second key-value pair feature, generating a second attention matrix, where the second attention matrix is ​​an attention matrix between the query video and the domain agent; Determine a target attention feature based on the first attention matrix and the second attention matrix, wherein the target attention feature is used to represent the attention of the support set on the query video; Target segmentation is performed based on the target attention feature to obtain a target segmentation result, and the target segmentation result is used to characterize the position of the target object in the video image of each frame.

2. The method according to claim 1, characterized in that The generating a first attention matrix based on the first key-value pair feature and the query feature includes: Performing feature sampling on the query feature to obtain the agent query feature corresponding to the domain agent; The first attention matrix is ​​generated based on a dot product result of the proxy query feature and the first key feature.

3. The method according to claim 1, characterized in that The generating a second attention matrix based on the query feature and the second key-value pair feature includes: Performing feature sampling on the second key feature to obtain a proxy key feature corresponding to the domain proxy; The second attention matrix is ​​generated based on the dot product result of the surrogate key feature and the query feature.

4. The method according to claim 1, characterized in that: The determining the target attention feature based on the first attention matrix and the second attention matrix includes: The target attention feature is obtained based on the matrix product result of the first attention matrix, the second attention matrix and the first value feature.

5. The method according to any one of claims 1 to 4, characterized in that: The target segmentation is performed based on the target attention feature to obtain a target segmentation result, including: Performing feature concatenation on the target attention feature and the second value feature to obtain a target feature of the query video; The target features are upsampled and convolved by a decoder to obtain a segmentation mask corresponding to each frame of the video image, wherein the segmentation mask is a binary mask, and the pixel value in the segmentation mask is used to indicate whether the corresponding pixel point belongs to the target object.

6. The method according to any one of claims 1 to 4, characterized in that: The step of extracting features from the support image in the support set to obtain target object features of the target object in the support image includes: Extracting features of the support image in the support set by an encoder to obtain support image features of the support image; The target object feature of the target object is obtained based on the supporting image feature and a marked mask corresponding to the supporting image, wherein the marked mask is used to indicate the position of the target object in the supporting image.

7. A video object segmentation method, characterized in that: The method comprises: Performing feature extraction on a first sample support image in a first sample support set to obtain a first sample object feature of a first sample object in the first sample support image; Extracting features of a sample video image in a sample query video to obtain sample video image features of the sample video image; Mapping the first sample object feature to a first sample key-value pair feature through a first convolutional layer of an attention model, where the first sample key-value pair feature consists of a first sample key feature and a first sample value feature; Mapping the sample video image feature into a sample query feature and a second sample key-value pair feature through a second convolutional layer of the attention model, wherein the second sample key-value pair feature is composed of a second sample key feature and a second sample value feature; Based on the first sample key-value pair feature and the sample query feature, generating a first sample attention matrix, wherein the first sample attention matrix is ​​an attention matrix between the first sample support set and a sample domain proxy, and the sample domain proxy is a frame of sample video image in the sample query video; Based on the sample query feature and the second sample key-value pair feature, generating the second sample attention matrix, where the second sample attention matrix is ​​an attention matrix between the sample query video and the sample domain agent; Determine a sample attention feature based on the first sample attention matrix and the second sample attention matrix, where the sample attention feature is used to represent the attention of the first sample support set on the sample query video; Based on the sample attention feature, target segmentation is performed by a decoder to obtain a sample segmentation result, wherein the sample segmentation result is used to characterize the position of the first sample object in each frame of the sample video image; The attention model and the decoder are trained based on the sample segmentation result and the sample annotation information corresponding to the sample video image.

8. The method according to claim 7, characterized in that The sample segmentation result is a sample segmentation mask, the sample annotation information is a sample annotation mask, and the sample annotation mask is used to indicate the position of the sample object in the sample video image; The training of the attention model and the decoder based on the sample segmentation result and the sample annotation information corresponding to the sample video image includes: Determining a segmentation loss based on a cross entropy loss and an intersection-over-union loss between a sample segmentation mask and the sample annotation mask; The attention model and the decoder are trained based on the segmentation loss.

9. The method according to claim 7 or 8, characterized in that: The first sample object feature and the sample video image feature are extracted by an encoder; The method further comprises: During the test, the encoder is used to extract features from a second sample support image in a second sample support set to obtain second sample object features of a second sample object in the sample support image; Selecting a sampling support image from the second sample support image, and performing feature extraction on the sampling support image by the encoder to obtain sampling image features of the sampling support image; Inputting the second sample object feature and the sampled image feature into the attention model to obtain a sampled attention feature, where the sampled attention feature is used to represent the attention of the second sample support set on the sampled support image; Based on the sampling attention feature, performing target segmentation through the decoder to obtain a sampling segmentation result, wherein the sampling segmentation result is used to characterize the position of the second sample object in the sampling support image; The encoder is trained based on the sampling segmentation result and the support image annotation information corresponding to the sampled support image.

10. A video object segmentation device, characterized in that: The device comprises: A first extraction module is used to extract features from a support image in a support set to obtain target object features of a target object in the support image; A second extraction module is used to extract features of the video image in the query video to obtain video image features of the video image; A first matrix determination module, used for mapping the target object feature into a first key-value pair feature through a first convolutional layer, wherein the first key-value pair feature is composed of a first key feature and a first value feature; Mapping the video image feature into a query feature and a second key-value pair feature through a second convolutional layer, where the second key-value pair feature consists of a second key feature and a second value feature; Based on the first key-value pair feature and the query feature, generating a first attention matrix, wherein the first attention matrix is ​​an attention matrix between the support set and the domain agent, and the domain agent is a frame of video image in the query video; Based on the query feature and the second key-value pair feature, generating a second attention matrix, where the second attention matrix is ​​an attention matrix between the query video and the domain agent; A first feature determination module, configured to determine a target attention feature based on the first attention matrix and the second attention matrix, wherein the target attention feature is used to represent the attention of the support set on the query video; The first segmentation module is used to perform target segmentation based on the target attention feature to obtain a target segmentation result, and the target segmentation result is used to characterize the position of the target object in the video image of each frame.

11. A video object segmentation device, characterized in that: The device comprises: a third extraction module, configured to perform feature extraction on a first sample support image in a first sample support set to obtain a first sample object feature of a first sample object in the first sample support image; A fourth extraction module, used for extracting features of a sample video image in a sample query video to obtain sample video image features of the sample video image; A second feature determination module, configured to map the first sample object feature to a first sample key-value pair feature through a first convolutional layer of an attention model, wherein the first sample key-value pair feature is composed of a first sample key feature and a first sample value feature; Mapping the sample video image feature into a sample query feature and a second sample key-value pair feature through a second convolutional layer of the attention model, wherein the second sample key-value pair feature is composed of a second sample key feature and a second sample value feature; Based on the first sample key-value pair feature and the sample query feature, generating a first sample attention matrix, wherein the first sample attention matrix is ​​an attention matrix between the first sample support set and a sample domain proxy, and the sample domain proxy is a frame of sample video image in the sample query video; Based on the sample query feature and the second sample key-value pair feature, generating the second sample attention matrix, where the second sample attention matrix is ​​an attention matrix between the sample query video and the sample domain agent; Determine a sample attention feature based on the first sample attention matrix and the second sample attention matrix, where the sample attention feature is used to represent the attention of the first sample support set on the sample query video; A second segmentation module is used to perform target segmentation through a decoder based on the sample attention feature to obtain a sample segmentation result, wherein the sample segmentation result is used to characterize the position of the first sample object in each frame of the sample video image; The first training module is used to train the attention model and the decoder based on the sample segmentation result and the sample annotation information corresponding to the sample video image.

12. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the video target segmentation method as described in any one of claims 1 to 6, or to implement the video target segmentation method as described in any one of claims 7 to 9.

13. A computer-readable storage medium, characterized in that: The readable storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the video target segmentation method as described in any one of claims 1 to 6, or to implement the video target segmentation method as described in any one of claims 7 to 9.

Images