Method, device, equipment and storage medium for object segmentation of video images

Abstract

The embodiments of the present application provide a method, device, equipment and storage medium for object segmentation of video images, and relate to the field of image processing technology. The method calculates a test feature vector and a reference feature vector, and then obtains a target feature vector from a time feature database, and uses a cascade of multiple multi-level feature association modules to extract features from the target feature vector, the reference feature vector and the test feature vector to obtain an object feature matrix, and based on the time feature database, determines the target feature value of the object feature matrix under each object feature, and decodes the target feature value to obtain a target mask result. Decoupling the self-attention process into three sub-streams can capture tiny targets and avoid ignoring important features during feature extraction. By associating the information extracted from historical video frames with the time feature database, long-term modeling is performed based on time series to better capture spatiotemporal interaction information and enhance the accuracy of the segmentation results of object features.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device, equipment and storage medium for object segmentation of video images. Background Art

[0002] Video Object Segmentation (VOS) is to extract objects of interest from a video sequence frame by frame. When performing segmentation, temporal continuity and motion information must be considered. It has a wide range of applications in multiple scenarios, such as video editing, augmented reality, surveillance video analysis, etc.

[0003] In the related art, video object segmentation mainly relies on a matching-based framework, which performs temporal feature matching after feature extraction. However, this approach does not work well for small and faint objects because fine-grained detail information may be ignored during feature extraction. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a method, device, equipment and storage medium for object segmentation of video images, so as to improve the segmentation accuracy of small targets in the process of video object segmentation.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a method for object segmentation of a video image, comprising:

[0006] Obtaining a test video frame in a target video, selecting a reference video frame from all historical video frames before the test video frame, and calculating a test feature vector of the test video frame and a reference feature vector of the reference video frame;

[0007] Acquire a feature value corresponding to at least one object feature from a time feature database, and perform feature extraction on the feature value to obtain a target feature vector, wherein the feature value is used to indicate a position of the object feature in at least one of the historical video frames;

[0008] Using a plurality of cascaded multi-level feature association modules to perform feature extraction on the target feature vector, the reference feature vector and the test feature vector to obtain an object feature matrix, the multi-level feature association module calculates an object self-attention matrix, a reference object enhancement matrix and a test reference correlation matrix according to three different input matrices, and inputs the object self-attention matrix, the reference object enhancement matrix and the test reference correlation matrix into a feature extraction module to obtain three different output matrices, the output matrix corresponding to the test reference correlation matrix of the last multi-level feature association module being the object feature matrix;

[0009] Based on the time feature database, a target feature value of the object feature matrix under each object feature is determined, and the target feature value is decoded to obtain a target mask result, wherein the target mask result includes a segmentation result of each object feature.

[0010] In some embodiments, before acquiring a feature value corresponding to at least one object feature from the time feature database, the method further includes:

[0011] Acquire a historical mask result of the historical video frame, wherein the historical mask result includes mask data corresponding to each of the object features, and the mask data is used to indicate a position of the object feature in the historical video frame;

[0012] For each of the object features, the mask data is spliced ​​with the historical video frame and then feature encoded to obtain a historical feature value, and the historical feature values ​​are fused in time sequence to obtain the feature value.

[0013] In some embodiments, the process of the multi-level feature association module calculating the object self-attention matrix, the reference object enhancement matrix and the test reference correlation matrix according to three different input matrices comprises the following steps:

[0014] Calculate the object self-attention matrix based on the first input matrix;

[0015] Calculate a reference object enhancement matrix based on the first input matrix and the second input matrix;

[0016] Calculating a test-reference correlation matrix based on the second input matrix and the third input matrix;

[0017] The three input matrices of the first multi-level feature association module are respectively the target feature vector, the reference feature vector and the test feature vector.

[0018] In some embodiments, calculating the object self-attention matrix according to the first input matrix includes:

[0019] Inputting the first input matrix into a first attention layer to obtain a first query vector, a first key vector, and a first value vector;

[0020] The object self-attention matrix is ​​calculated using a softmax function based on the first query vector, the first key vector and the first value vector.

[0021] In some embodiments, the step of calculating a reference object enhancement matrix based on the first input matrix and the second input matrix comprises:

[0022] Inputting the first input matrix and the second input matrix into a second attention layer to obtain a second query vector;

[0023] Calculate a first intermediate value corresponding to the target feature vector and a second intermediate value corresponding to the reference feature vector using a projection matrix of a second key vector corresponding to the second attention layer, and connect the first intermediate value and the second intermediate value to obtain a second key vector;

[0024] Calculate a third intermediate value corresponding to the target feature vector and a fourth intermediate value corresponding to the reference feature vector using a projection matrix of the second value vector corresponding to the second attention layer, and connect the third intermediate value and the fourth intermediate value to obtain a second value vector;

[0025] The reference object enhancement matrix is ​​calculated according to the second query vector, the second key vector and the second value vector using a softmax function.

[0026] In some embodiments, the feature extraction module includes a cascaded feedforward neural network, a normalization layer, and a multilayer perceptron, and the object self-attention matrix, the reference object enhancement matrix, and the test reference correlation matrix are input into the feature extraction module to obtain three different output matrices, including:

[0027] The object self-attention matrix, the reference object enhancement matrix and the test reference correlation matrix are respectively used as intermediate input matrices;

[0028] Inputting the intermediate input matrix into the feedforward neural network for feature extraction to obtain a first feature;

[0029] Inputting the first feature into the normalization layer for feature extraction to obtain a second feature;

[0030] The second feature is input into the multi-layer perceptron for feature extraction to obtain the output matrix.

[0031] In some embodiments, determining the target feature value of the object feature matrix under each object feature based on the time feature database includes:

[0032] Acquire the feature value of each of the object features in the time feature database, and segment each of the historical feature values ​​from the feature value;

[0033] The similarity between each of the historical feature values ​​and the object feature matrix is ​​calculated, and the target feature value corresponding to the object feature is selected from the historical feature values ​​according to the similarity.

[0034] In some embodiments, the decoding process of the target feature value to obtain a target mask result includes:

[0035] Obtaining an overall correlation feature according to the target feature value of each of the object features;

[0036] The overall associated features are input into a decoder for decoding processing to obtain the target mask result.

[0037] In some embodiments, selecting a reference video frame from all historical video frames before the test video frame includes:

[0038] Selecting the first historical video frame as the first reference video frame in time sequence;

[0039] Based on a preset selection rule, at least one of the other historical video frames is selected as the other reference video frame.

[0040] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a video image object segmentation device, comprising:

[0041] The first feature extraction module is used to obtain a test video frame in the target video, select a reference video frame from all historical video frames before the test video frame, and calculate a test feature vector of the test video frame and a reference feature vector of the reference video frame;

[0042] A second feature extraction module is used to obtain a feature value corresponding to at least one object feature from a time feature database, and perform feature extraction on the feature value to obtain a target feature vector, wherein the feature value is used to indicate a position of the object feature in at least one of the historical video frames;

[0043] The third feature extraction module is used to perform feature extraction on the target feature vector, the reference feature vector and the test feature vector by using a plurality of cascaded multi-level feature association modules to obtain an object feature matrix. The multi-level feature association module calculates an object self-attention matrix, a reference object enhancement matrix and a test reference correlation matrix according to three different input matrices, and inputs the object self-attention matrix, the reference object enhancement matrix and the test reference correlation matrix into a feature extraction module to obtain three different output matrices. The output matrix corresponding to the test reference correlation matrix of the last multi-level feature association module is the object feature matrix.

[0044] Target segmentation module: used to determine the target feature value of the object feature matrix under each object feature based on the time feature database, decode the target feature value to obtain a target mask result, and the target mask result includes the segmentation result of each object feature.

[0045] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0046] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0047] The object segmentation method, apparatus, device and storage medium of a video image proposed in the embodiments of the present application obtain a test video frame in a target video, select a reference video frame from all historical video frames before the test video frame, calculate a test feature vector of the test video frame and a reference feature vector of the reference video frame; obtain a feature value corresponding to at least one object feature from a temporal feature database, and perform feature extraction on the feature value to obtain a target feature vector, wherein the feature value is used to indicate the position of the object feature in at least one historical video frame; perform feature extraction on the target feature vector, the reference feature vector and the test feature vector using a cascade of multiple multi-level feature association modules to obtain an object feature matrix, wherein the multi-level feature association module calculates an object self-attention matrix, a reference object enhancement matrix and a test reference correlation matrix according to three different input matrices, and inputs the object self-attention matrix, the reference object enhancement matrix and the test reference correlation matrix into a feature extraction module to obtain three different output matrices, wherein the output matrix corresponding to the test reference correlation matrix of the last multi-level feature association module is an object feature matrix; based on the temporal feature database, determine the target feature value of the object feature matrix under each object feature, perform decoding processing on the target feature value, and obtain a target mask result, wherein the target mask result includes the segmentation result of each object feature. In the embodiment of the present application, the self-attention process is decoupled into three sub-streams, namely object self-attention, reference object enhancement, and test reference correlation. This mechanism can capture small targets, thereby avoiding ignoring some important features in the feature extraction process. On the other hand, when performing object segmentation, the information extracted from the historical video frames is associated with the time feature database, and long-term modeling based on time series is performed to better capture the spatiotemporal interaction information corresponding to the object features, thereby enhancing the accuracy of the segmentation results of the object features. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flowchart of a method for object segmentation of a video image provided in an embodiment of the present application.

[0049] Figure 2It is a flowchart of selecting a reference video frame from all historical video frames before a test video frame provided by an embodiment of the present application.

[0050] Figure 3 It is a schematic diagram of the storage content of the time feature database provided in an embodiment of the present application.

[0051] Figure 4 It is a flow chart of calculating the feature matrix of an object provided in an embodiment of the present application.

[0052] Figure 5 It is a flowchart of calculating a reference object enhancement matrix based on a first input matrix and a second input matrix provided in an embodiment of the present application.

[0053] Figure 6 It is a schematic diagram of the structure of the feature extraction module in the embodiment of the present application.

[0054] Figure 7 It is a flowchart of determining the target feature value of the object feature matrix under each object feature based on the time feature database provided by an embodiment of the present application.

[0055] Figure 8 It is a schematic diagram of the feature values ​​of the updated object features provided in the embodiment of the present application.

[0056] Fig. 9 It is an overall flow chart of the object segmentation method of video images provided in an embodiment of the present application.

[0057] Fig.10 This is a structural block diagram of a device for object segmentation of video images provided by another embodiment of the present application.

[0058] Fig.11 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] It should be noted that although the functional modules are divided in the device schematic and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0062] First, some nouns involved in this application are analyzed:

[0063] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0064] Video Object Segmentation (VOS) is to extract objects of interest from a video sequence frame by frame. When performing segmentation, temporal continuity and motion information must be considered. It has a wide range of applications in multiple scenarios, such as video editing, augmented reality, surveillance video analysis, etc.

[0065] In the related art, video object segmentation mainly relies on a matching-based framework, which performs temporal feature matching after feature extraction. However, this approach does not work well for small and weak object segmentation because fine-grained detail information may be ignored during the feature extraction process. In addition, the video object segmentation algorithm in the related art has difficulties in processing long videos and has poor simulation performance for long-term associations, making it difficult to adapt to complex scenes.

[0066] Based on this, the embodiments of the present application provide a method, apparatus, device and storage medium for object segmentation of video images, by decoupling the self-attention process into three sub-streams, namely object self-attention, reference object enhancement and test reference correlation. Through this mechanism, small targets can be captured, thereby avoiding ignoring some important features in the feature extraction process. On the other hand, when performing object segmentation, the information extracted from the historical video frames is associated with the time feature database, and long-term modeling based on time series is performed to better capture the spatiotemporal interaction information corresponding to the object features, thereby enhancing the accuracy of the segmentation results of the object features.

[0067] The embodiments of the present application provide a method, apparatus, device and storage medium for object segmentation of video images, which are specifically described through the following embodiments. First, the method for object segmentation of video images in the embodiments of the present application is described.

[0068] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0069] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0070] The object segmentation method of the video image provided by the embodiment of the present application relates to the field of image processing technology. The object segmentation method of the video image provided by the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be a computer program running in the terminal or the server side. For example, the computer program can be a native program or software module in the operating system; it can be a local (Native) application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a client that supports object segmentation of video images, or it can be a small program, that is, a program that can be run only by downloading it to the browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module or plug-in. Among them, the terminal communicates with the server through a network. The object segmentation method of the video image can be executed by the terminal or the server, or by the terminal and the server in collaboration.

[0071] In some embodiments, the terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer, or a smart watch. In addition, the terminal may also be an intelligent vehicle-mounted device. The intelligent vehicle-mounted device applies the object segmentation method of the video image of this embodiment to provide related services and enhance the driving experience. The server may be an independent server, or it may be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms; it may also be a service node in a blockchain system, and each service node in the blockchain system forms a peer-to-peer (P2P, Peer To Peer, P2P) network. The P2P protocol is an application layer protocol running on the Transmission Control Protocol (TCP) protocol. The terminal and the server may be connected via Bluetooth, Universal Serial Bus (USB) or network communication connection methods, which are not limited in this embodiment.

[0072] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0073] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0074] The following describes a method for object segmentation of a video image in an embodiment of the present application.

[0075] Figure 1 is an optional flow chart of the method for object segmentation of a video image provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps 110 to 140. It can also be understood that this embodiment Figure 1 The order of step 110 to step 140 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0076] Step 110: Obtain a test video frame in the target video, select a reference video frame from all historical video frames before the test video frame, and calculate a test feature vector of the test video frame and a reference feature vector of the reference video frame.

[0077] In one embodiment, the target video is divided into multiple video frames in time sequence, and the video frames that need to be segmented are selected one by one from the video frames as test video frames, and the test video frames that have been segmented are used as historical video frames. It is understandable that the embodiment of the present application can select a part or all of the video frames as test video frames according to computing power and demand. The following embodiment is described by taking the object segmentation process of one of the test video frames as an example.

[0078] In one embodiment, for the historical video frames, a reference video frame needs to be selected from them to assist the object segmentation process of the test video frame. Figure 2 , Figure 2 The flowchart of selecting a reference video frame from all historical video frames before the test video frame provided by an embodiment of the present application specifically includes the following steps:

[0079] Step 210: Select the first historical video frame as the first reference video frame according to the time sequence.

[0080] In one embodiment, the historical video frames are updated in real time in the spatial memory in a first-in-first-out order, and in order to avoid the loss of object features in a long-term scene, the first historical video frame in the spatial memory will not be discarded during the update process. Therefore, when selecting a reference video frame, the first historical video frame of the target video must also be selected.

[0081] Step 220: Based on a preset selection rule, select at least one from other historical video frames as another reference video frame.

[0082] In one embodiment, after selecting the first historical video frame, in order to provide more spatial information to assist in the object segmentation process, it is necessary to select at least one reference video frame from other historical video frames. Here, the selection can be performed according to a preset selection rule, and the preset selection rule can be any one of the rules such as full selection, interval selection, random selection, etc., which is not limited in the embodiment of the present application.

[0083] According to the above process, multiple reference video frames are selected and obtained, and a reference feature vector corresponding to each reference video frame is obtained by embedding. At the same time, a test feature vector of a test video frame is obtained by embedding.

[0084] Step 120: Acquire a feature value corresponding to at least one object feature from a time feature database, and perform feature extraction on the feature value to obtain a target feature vector.

[0085] In one embodiment, the time feature database is used to store time memory features, and through time-series-based long-term modeling, it is possible to better capture the spatiotemporal interaction information corresponding to the object features, thereby enhancing the accuracy of the segmentation results of the object features.

[0086] In an embodiment of the present application, the temporal feature database stores the object features and their overall feature values ​​obtained by segmentation in each historical video frame. The number of object features is consistent with the number of objects to be segmented, and may involve multiple objects. For example, in a riding video, the object may be a bicycle, a rider, etc. In addition, there may be one object in the first historical video frame, and the object in the second historical video frame may be increased to two. The objects in all historical video frames are accumulated, and then the objects are digitized, for example, a number is first assigned to each object, and the object features are obtained by pooling the numbers. The same objects have the same object features. In addition, each object feature in the temporal feature database contains a feature value, which is used to indicate the position of the corresponding object feature in at least one historical video frame, and this position is represented by the historical feature value corresponding to the object feature. Due to the correlation between object features and feature values, it can also be understood by the concept of key-value, where the object feature is the key and the feature value is the corresponding value. The subsequent embodiments will describe the calculation process of the feature value in detail.

[0087] Reference Figure 3 , Figure 3 It is a schematic diagram of the storage content of the time feature database provided in an embodiment of the present application. Figure 3 In the figure, four object features are used as an example for explanation, where each object feature is distinguished by an ID, such as ID1, ID2, ID3, and ID4 shown in the figure. The positions related to different object features are obtained through each historical video frame, which are expressed as historical feature values, and then the historical feature values ​​of each object feature are spliced ​​as the feature value of the object feature.

[0088] It can be understood that, for the first test video frame, since there is no corresponding historical video frame, the feature value corresponding to the object feature at this time may be a default value, such as an all-zero matrix.

[0089] In one embodiment, the feature value of each object feature contained in the time feature database is obtained, and all the feature values ​​are concatenated and input into the projection layer for feature extraction to obtain the target feature vector. The projection layer is implemented by linear transformation and nonlinear activation function.

[0090] Step 130: Utilize a plurality of cascaded multi-level feature association modules to perform feature extraction on the target feature vector, the reference feature vector and the test feature vector to obtain an object feature matrix.

[0091] In one embodiment, referring to Figure 4 , Figure 4 It is a flow chart of calculating the feature matrix of an object provided in an embodiment of the present application. Figure 4 The multi-level feature association module has three different input matrices. After the three input matrices are aligned through the projection layer, correlation calculation is performed. That is, the object self-attention matrix, the reference object enhancement matrix and the test reference correlation matrix are calculated based on the three different input matrices.

[0092] Next, the object self-attention matrix, the reference object enhancement matrix and the test reference correlation matrix are respectively input into the feature extraction module to obtain three different output matrices. This is a parallel operation, which can also be understood as three feature extraction modules corresponding to different inputs.

[0093] Then use the three different output matrices as the input matrices of the next multi-level feature association module, and perform similar operations to obtain the three outputs of the feature extraction module of the last multi-level feature association module. Among them, the three outputs in the last multi-level feature association module are respectively associated with the object self-attention matrix, reference object enhancement matrix and test reference correlation matrix obtained in the module, and the output matrix corresponding to the test reference correlation matrix is ​​selected as the object feature matrix.

[0094] In one embodiment, since it is inefficient to directly calculate the cross attention between the three inputs, the self-attention process is decoupled into three sub-streams after alignment to perform different correlation calculations. Figure 4 ,The process of calculating the object self-attention matrix, the reference object enhancement matrix and the test reference correlation matrix according to three different input matrices by the multi-level feature association module specifically includes: calculating the object self-attention matrix according to the first input matrix, calculating the reference object enhancement matrix according to the first input matrix and the second input matrix, and calculating the test reference correlation matrix according to the second input matrix and the third input matrix. Among them, the three input matrices of the first multi-level feature association module are the target feature vector, the reference feature vector and the test feature vector.

[0095] Taking the first multi-level feature association module as an example, the process of correlation calculation is described as follows: object self-attention matrix calculation is implemented based on the target feature vector, reference object enhancement matrix calculation is implemented based on the target feature vector and the reference feature vector, and test reference correlation matrix calculation is implemented based on the reference feature vector and the test feature vector.

[0096] In one embodiment, for a first input matrix, such as a target feature vector, self-attention is performed to enhance the distinguishing information between different objects. The process of calculating the object self-attention matrix according to the first input matrix specifically includes: inputting the first input matrix into the first attention layer to obtain a first query vector, a first key vector and a first value vector; using a softmax function to calculate the object self-attention matrix according to the first query vector, the first key vector and the first value vector.

[0097] Taking the first multi-level feature association module as an example, the object self-attention matrix is ​​calculated based on the target feature vector. The obtained object self-attention matrix It is expressed as:

[0098]

[0099] in, represents the first query vector, represents the first key vector, represents the first value vector, represents the softmax function, d represents the dimension of the first key vector, and is the scaling factor.

[0100] The first attention layer above uses the self-attention mechanism to project the target feature vector and convert it into the first query vector, the first key vector, and the first value vector. Specifically, three different linear transformations are applied to the target feature vector through matrix multiplication, expressed as:

[0101]

[0102]

[0103]

[0104] in, represents the target feature vector, represents the projection matrix of the first query vector, represents the projection matrix of the first bond vector, Represents the projection matrix of the first value vector.

[0105] It can be understood that the above projection matrix is ​​determined through a pre-performed training process.

[0106] Next, the target feature vector is used to enhance each reference feature vector by associating the target feature vector with each reference feature vector, and a reference object enhancement matrix is ​​calculated based on the first input matrix and the second input matrix. Figure 5 , Figure 5 This is a flowchart of calculating a reference object enhancement matrix according to a first input matrix and a second input matrix provided by an embodiment of the present application, which specifically includes the following steps:

[0107] Step 510: Input the first input matrix and the second input matrix into the second attention layer to obtain a second query vector.

[0108] Step 520: Use the projection matrix of the second key vector corresponding to the second attention layer to calculate the first intermediate value corresponding to the target eigenvector and the second intermediate value corresponding to the reference eigenvector, and connect the first intermediate value and the second intermediate value to obtain the second key vector.

[0109] Step 530: Calculate the third intermediate value corresponding to the target eigenvector and the fourth intermediate value corresponding to the reference eigenvector using the projection matrix of the second value vector corresponding to the second attention layer, and connect the third intermediate value and the fourth intermediate value to obtain the second value vector.

[0110] Step 540: Utilize the softmax function to calculate a reference object enhancement matrix according to the second query vector, the second key vector and the second value vector.

[0111] In one embodiment, taking the first multi-level feature association module as an example, the reference object enhancement matrix is ​​calculated based on the target feature vector and the reference feature vector. Referring to the first attention layer, the second attention layer also obtains three projection matrices through pre-training, which are the projection matrices of the second query vector , the projection matrix of the second key vector and the projection matrix of the second value vector .

[0112] Among them, the second query vector is expressed as:

[0113]

[0114] in, represents the i-th reference eigenvector.

[0115] The second key vector is represented as:

[0116]

[0117] in, represents the i-th reference eigenvector, represents the second key vector corresponding to the i-th reference eigenvector, represents the first intermediate value, represents the second intermediate value, represents the connection function, and [] represents matrix multiplication.

[0118] The second value vector is represented as:

[0119]

[0120] in, represents the second value vector corresponding to the i-th reference eigenvector, represents the third intermediate value, Indicates the fourth intermediate value.

[0121] The reference object enhancement matrix is ​​expressed as:

[0122]

[0123] in, represents the reference object enhancement matrix corresponding to the i-th reference eigenvector, Indicates the dimension of the second key vector.

[0124] It is understandable that the basic principle of segmenting each reference feature vector separately during information association is to ensure that the reference feature vectors are not affected by each other and maintain their discrete integrity. After obtaining multiple reference object enhancement matrices, they can be spliced ​​to obtain an overall reference object enhancement matrix to apply the information corresponding to all reference feature vectors.

[0125] In one embodiment, a test reference correlation matrix is ​​calculated based on the second input matrix and the third input matrix. Taking the first multi-level feature association module as an example, in order to perform asymmetric information propagation between the reference feature vector and the test feature vector, so that the test feature vector only receives relevant information of the reference feature vector, while the reference feature vector remains unchanged, the test reference correlation matrix is ​​calculated based on the reference feature vector and the test feature vector in combination with the calculation process of the reference object enhancement matrix. For example, the third query vector is calculated using the third attention layer, the third key vector is calculated using the projection matrix of the third key vector corresponding to the third attention layer, and the third value vector is calculated using the projection matrix of the third value vector corresponding to the third attention layer; finally, the test reference correlation matrix is ​​calculated based on the third query vector, the third key vector and the third value vector using the softmax function.

[0126] The process is expressed as:

[0127]

[0128]

[0129]

[0130]

[0131] in, represents the projection matrix of the third query vector, represents the projection matrix of the third key vector, represents the projection matrix of the third value vector, represents the third key vector, represents the third value vector, represents the test feature vector, represents the reference feature vector for concatenation, represents the test-reference correlation matrix, Indicates the dimension of the third key vector.

[0132] In one embodiment, referring to Figure 6 , Figure 6It is a schematic diagram of the feature extraction module structure in the embodiment of the present application. The feature extraction module includes a cascaded feedforward neural network ((Feedforward Neural Network, FFN), a normalization layer (LayerNormalization, LN) and a multilayer perceptron (Multilayer Perceptron, MLP). Specifically, the feedforward neural network FFN contains two linear transformation layers, with a nonlinear activation function such as ReLU or GELU in the middle. The first linear layer maps the input to a hidden layer, and the activation function introduces nonlinearity to help the network capture complex features; the second linear layer maps the hidden layer output back to the original dimension. The normalization layer (LN) normalizes the input data to have zero mean and unit variance, and then adjusts the normalized data through scaling and offset parameters to help stabilize the gradient and prevent the gradient from disappearing or exploding. The multilayer perceptron MLP consists of multiple fully connected layers, each of which uses a nonlinear activation function such as Sigmoid, Tanh or ReLU.

[0133] Therefore, combined Figure 6 The process of inputting the object self-attention matrix, the reference object enhancement matrix and the test reference correlation matrix into the feature extraction module to obtain three different output matrices specifically includes: using the object self-attention matrix, the reference object enhancement matrix and the test reference correlation matrix as intermediate input matrices respectively, inputting the intermediate input matrices into the feedforward neural network for feature extraction to obtain the first feature, inputting the first feature into the normalization layer for feature extraction to obtain the second feature, and inputting the second feature into the multilayer perceptron for feature extraction to obtain the output matrix.

[0134] In one embodiment, assuming that the intermediate input matrix is ​​F, the feedforward neural network FFN processes the intermediate input matrix through a series of linear transformations and nonlinear activation functions (such as ReLU), and the extracted first feature F1 is expressed as: F1=FNN(F). Next, the normalization layer LN standardizes the first feature F1, and the obtained second feature F2 is expressed as: F2=LN(F1). Finally, the multilayer perceptron MLP uses the fully connected layer and the nonlinear activation function between each layer to further extract features from the second feature F2, and generates the final output matrix Out, expressed as: Out=MLP(F2). The feature extraction network is used to extract useful information from the intermediate input matrix.

[0135] Combination Figure 4, the three outputs of the feature extraction network of the last multi-level feature association module correspond to its object self-attention matrix, reference object enhancement matrix and test reference correlation matrix respectively, and the output matrix corresponding to the test reference correlation matrix is ​​selected as the object feature matrix. Due to the cascade process, the calculation of the test reference correlation matrix obtained by the last multi-level feature association module is related to all the previous related features, which can indicate the interaction data between different types of inputs, that is, the obtained object feature matrix can obtain relatively complete spatiotemporal interaction information.

[0136] Step 140: Based on the time feature database, determine the target feature value of the object feature matrix under each object feature, decode the target feature value, and obtain a target mask result.

[0137] In one embodiment, the target mask result includes the segmentation result of each object feature. Figure 3 , each object feature's feature value includes at least one historical feature value, so for each object feature, it is necessary to select the closest one from the historical feature values ​​as the target feature value, and this process is called ID association. Figure 7 , Figure 7 This is a flowchart of determining the target feature value of the object feature matrix under each object feature based on the time feature database provided by an embodiment of the present application, which specifically includes the following steps:

[0138] Step 710: Acquire the feature value of each object feature in the time feature database, and segment each historical feature value from the feature value.

[0139] In one embodiment, there are as many historical feature values ​​as there are historical video frames.

[0140] Step 720: Calculate the similarity between each historical feature value and the object feature matrix, and select the target feature value corresponding to the object feature from the historical feature values ​​according to the similarity.

[0141] In one embodiment, the similarity between each historical feature value and the object feature matrix is ​​calculated, the maximum value is selected from all similarities, and the historical feature value corresponding to the maximum similarity is used as the target feature value of the object feature. Finally, the target feature value of each object feature is obtained in turn. Figure 3 As can be seen from the target feature values ​​shown in the bold box, target feature values ​​of different object features may correspond to different historical video frames.

[0142] In one embodiment, the process of decoding the target feature value to obtain the target mask result specifically includes: obtaining the overall correlation feature according to the target feature value of each object feature, for example, by splicing to obtain the overall correlation feature. Since the historical feature value is related to the object in the historical video frame, the overall correlation feature contains the position information corresponding to each object feature, and the overall correlation feature is input into the decoder for decoding to obtain the target mask result. The decoder here is used to convert the overall correlation feature into a pixel-level segmentation mask, specifically upsampling the overall correlation feature to match the resolution of the test video frame, and classifying pixel by pixel to predict the category of each pixel, thereby obtaining the target mask result. Among them, the decoder can be a U-Net architecture.

[0143] In one embodiment, after obtaining the target mask result, the current test video frame becomes a historical video frame, and the historical feature value of each object feature is calculated based on its target mask result. The specific process is described as follows: obtaining the historical mask result of the historical video frame, the historical mask result is the target mask data of the historical video frame as the reference video frame, and since the target mask data includes the mask results corresponding to different objects, the mask data corresponding to each object feature can be obtained therefrom, and the mask data is used to indicate the position of the object feature in the historical video frame. Next, for each object feature, the mask data is spliced ​​with the historical video frame and feature encoding is performed to obtain the historical feature value, and then the feature value is obtained by fusing the historical feature value in time sequence.

[0144] In one embodiment, referring to Figure 8 , Figure 8 It is a schematic diagram of the feature values ​​of the updated object features provided in the embodiment of the present application. Figure 8 In the process, the historical mask results of the historical video frames and the corresponding historical video frames are spliced ​​and input into the memory encoder after splicing. The encoder can extract features from the splicing results to obtain historical feature values, which can indicate the object position obtained by combining the historical mask results and the historical video frames.

[0145] It is understandable that if there is no historical video frame in the time feature database, the feature value at this time is the default value. When the first historical feature value is obtained, the feature value becomes the historical feature value. If there is more than one historical feature value, fusion is performed. The fusion process is to splice all historical feature values ​​to obtain the total feature value.

[0146] In one embodiment, considering that the memory space of the time feature database is limited, if the target video is long, the memory may be full after the historical video frames are continuously updated, so it is necessary to manage the memory and remove some historical video frames to make new space. In this embodiment, a counter is allocated to each historical video frame to count the frequency of its corresponding historical feature value being selected as the target feature value. When the memory threshold is reached, the historical video frame with the lowest frequency is selected according to the frequency corresponding to the counter, and all its corresponding historical feature values ​​are removed from the time feature database.

[0147] In one embodiment, referring to Fig. 9 , Fig. 9 It is an overall flow chart of the object segmentation method of video images provided in an embodiment of the present application.

[0148] Combination Fig. 9 It can be seen that the test video frame in the target video is obtained, and the reference video frame is selected from all the historical video frames before the test video frame, and then the test feature vector of the test video frame and the reference feature vector of the reference video frame are calculated using the embedding layer. The feature values ​​corresponding to multiple object features are obtained from the time feature database, and the feature values ​​are extracted using the projection layer to obtain the target feature vector.

[0149] Next, multiple cascaded multi-level feature association modules are used to extract features from the target feature vector, reference feature vector and test feature vector to obtain the object feature matrix. Among them, the multi-level feature association module calculates the object self-attention matrix, the reference object enhancement matrix and the test reference correlation matrix according to three different input matrices. Taking the first multi-level feature association module as an example, the process of correlation calculation is described as follows: the object self-attention matrix is ​​calculated based on the target feature vector, the reference object enhancement matrix is ​​calculated based on the target feature vector and the reference feature vector, and the test reference correlation matrix is ​​calculated based on the reference feature vector and the test feature vector.

[0150] Then, the object self-attention matrix, reference object enhancement matrix and test reference correlation matrix obtained in the multi-level feature association module are input into the feature extraction module to obtain three different output matrices. Among them, the feature extraction module includes a cascaded feedforward neural network, a normalization layer and a multi-layer perceptron. The output matrix corresponding to the test reference correlation matrix of the last multi-level feature association module is the object feature matrix. Next, based on the time feature database, ID association is performed to determine the target feature value of the object feature matrix under each object feature, and then the decoder is used to decode the target feature value to obtain the target mask result.

[0151] Finally, the test video frame is used as a historical video frame to update the temporal feature database. Specifically, the historical mask result of the historical video frame is spliced ​​with the corresponding historical video frame, and then the spliced ​​historical feature value is input into the memory encoder to obtain the historical feature value, and then the historical feature value is fused to obtain the feature value.

[0152] The object segmentation method of the video image provided by the embodiment of the present application decouples the self-attention process into three sub-streams, namely object self-attention, reference object enhancement and test reference correlation, and combines the test feature vector, reference feature vector and target feature vector to achieve a full interactive process. Through this mechanism, tiny targets can be captured, thereby avoiding ignoring some important features in the feature extraction process. When performing object segmentation, the temporal feature database is used to realize the parallel association of object features. Since the temporal feature database retains the information related to the object in the previous video frame, it can perform long-term modeling based on time series to achieve the purpose of matching, separating and enhancing the relevant features of the object in the test video frame, so as to better capture the spatiotemporal interaction information corresponding to the object features, thereby more effectively processing small targets and complex scenes, while meeting the spatial accuracy and temporal coherence, and improving the accuracy of object segmentation.

[0153] The present application also provides a video image object segmentation device, which can implement the above-mentioned video image object segmentation method, referring to Fig.10 , the device comprises:

[0154] The first feature extraction module 1010 is used to obtain a test video frame in the target video, select a reference video frame from all historical video frames before the test video frame, and calculate a test feature vector of the test video frame and a reference feature vector of the reference video frame.

[0155] The second feature extraction module 1020 is used to obtain a feature value corresponding to at least one object feature from the time feature database, and perform feature extraction on the feature value to obtain a target feature vector, wherein the feature value is used to indicate the position of the object feature in at least one historical video frame.

[0156] The third feature extraction module 1030 is used to perform feature extraction on the target feature vector, the reference feature vector and the test feature vector using a plurality of cascaded multi-level feature association modules to obtain an object feature matrix. The multi-level feature association module calculates the object self-attention matrix, the reference object enhancement matrix and the test reference correlation matrix according to three different input matrices, and inputs the object self-attention matrix, the reference object enhancement matrix and the test reference correlation matrix into the feature extraction module to obtain three different output matrices. The output matrix corresponding to the test reference correlation matrix of the last multi-level feature association module is the object feature matrix.

[0157] The target segmentation module 1040 is used to determine the target feature value of the object feature matrix under each object feature based on the time feature database, decode the target feature value, and obtain a target mask result, which includes the segmentation result of each object feature.

[0158] The specific implementation of the video image object segmentation device of this embodiment is basically the same as the specific implementation of the video image object segmentation method described above, and will not be described in detail here.

[0159] The present application also provides an electronic device, including:

[0160] at least one memory;

[0161] at least one processor;

[0162] at least one program;

[0163] The program is stored in the memory, and the processor executes the at least one program to implement the object segmentation method of the video image implemented in the present application. The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0164] See also Fig.11 , Fig.11 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0165] The processor 1101 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0166] The memory 1102 may be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1102 may store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1102, and the processor 1101 calls and executes the object segmentation method of the video image in the embodiments of this application;

[0167] Input / output interface 1103, used to implement information input and output;

[0168] Communication interface 1104, used to realize communication interaction between the device and other devices, which can be realized by wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.); and

[0169] A bus 1105 that transmits information between various components of the device (e.g., the processor 1101, the memory 1102, the input / output interface 1103, and the communication interface 1104);

[0170] The processor 1101 , the memory 1102 , the input / output interface 1103 and the communication interface 1104 are connected to each other in communication within the device via the bus 1105 .

[0171] An embodiment of the present application further provides a storage medium, which is a storage medium storing a computer program, and when the computer program is executed by a processor, the above-mentioned object segmentation method for video images is implemented.

[0172] As a non-transitory storage medium, the memory can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0173] The object segmentation method, apparatus, device and storage medium of a video image proposed in the embodiments of the present application obtain a test video frame in a target video, select a reference video frame from all historical video frames before the test video frame, calculate a test feature vector of the test video frame and a reference feature vector of the reference video frame; obtain a feature value corresponding to at least one object feature from a temporal feature database, and perform feature extraction on the feature value to obtain a target feature vector, wherein the feature value is used to indicate the position of the object feature in at least one historical video frame; perform feature extraction on the target feature vector, the reference feature vector and the test feature vector using a cascade of multiple multi-level feature association modules to obtain an object feature matrix, wherein the multi-level feature association module calculates an object self-attention matrix, a reference object enhancement matrix and a test reference correlation matrix according to three different input matrices, and inputs the object self-attention matrix, the reference object enhancement matrix and the test reference correlation matrix into a feature extraction module to obtain three different output matrices, wherein the output matrix corresponding to the test reference correlation matrix of the last multi-level feature association module is an object feature matrix; based on the temporal feature database, determine the target feature value of the object feature matrix under each object feature, perform decoding processing on the target feature value, and obtain a target mask result, wherein the target mask result includes the segmentation result of each object feature. In the embodiment of the present application, the self-attention process is decoupled into three sub-streams, namely object self-attention, reference object enhancement, and test reference correlation. This mechanism can capture small targets, thereby avoiding ignoring some important features in the feature extraction process. On the other hand, when performing object segmentation, the information extracted from the historical video frames is associated with the time feature database, and long-term modeling based on time series is performed to better capture the spatiotemporal interaction information corresponding to the object features, thereby enhancing the accuracy of the segmentation results of the object features.

[0174] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0175] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0176] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0177] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0178] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0179] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0180] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0181] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0182] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0183] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store programs.

[0184] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for object segmentation of a video image, characterized in that: include: Obtaining a test video frame in a target video, selecting a reference video frame from all historical video frames before the test video frame, and calculating a test feature vector of the test video frame and a reference feature vector of the reference video frame; Acquire a feature value corresponding to at least one object feature from a time feature database, and perform feature extraction on the feature value to obtain a target feature vector, wherein the feature value is used to indicate a position of the object feature in at least one of the historical video frames; Using a plurality of cascaded multi-level feature association modules to perform feature extraction on the target feature vector, the reference feature vector and the test feature vector to obtain an object feature matrix, the multi-level feature association module calculates an object self-attention matrix, a reference object enhancement matrix and a test reference correlation matrix according to three different input matrices, and inputs the object self-attention matrix, the reference object enhancement matrix and the test reference correlation matrix into a feature extraction module to obtain three different output matrices, the output matrix corresponding to the test reference correlation matrix of the last multi-level feature association module being the object feature matrix; Based on the time feature database, determine the target feature value of the object feature matrix under each of the object features, decode the target feature value to obtain a target mask result, wherein the target mask result includes a segmentation result of each of the object features; The process of the multi-level feature association module calculating the object self-attention matrix, the reference object enhancement matrix and the test reference correlation matrix according to three different input matrices comprises the following steps: Calculate the object self-attention matrix based on the first input matrix; Calculate a reference object enhancement matrix based on the first input matrix and the second input matrix; Calculating a test-reference correlation matrix based on the second input matrix and the third input matrix; The three input matrices of the first multi-level feature association module are the target feature vector, the reference feature vector and the test feature vector, respectively, and the three input matrices of other multi-level feature association modules are the three output matrices obtained by the feature extraction module of the previous multi-level feature association module based on the object self-attention matrix, the reference object enhancement matrix and the test reference correlation matrix.

2. The object segmentation method of video images according to claim 1, characterized in that: Before acquiring a feature value corresponding to at least one object feature from the time feature database, the method further includes: Acquire a historical mask result of the historical video frame, wherein the historical mask result includes mask data corresponding to each of the object features, and the mask data is used to indicate a position of the object feature in the historical video frame; For each of the object features, the mask data is spliced ​​with the historical video frame and then feature encoded to obtain a historical feature value, and the historical feature values ​​are fused in time sequence to obtain the feature value.

3. The object segmentation method of video images according to claim 1, characterized in that: The calculating the object self-attention matrix according to the first input matrix comprises: Inputting the first input matrix into a first attention layer to obtain a first query vector, a first key vector, and a first value vector; The object self-attention matrix is ​​calculated using a softmax function based on the first query vector, the first key vector and the first value vector.

4. The object segmentation method of video images according to claim 1, characterized in that: The step of calculating a reference object enhancement matrix according to the first input matrix and the second input matrix comprises: Inputting the first input matrix and the second input matrix into a second attention layer to obtain a second query vector; Calculate a first intermediate value corresponding to the target feature vector and a second intermediate value corresponding to the reference feature vector using a projection matrix of a second key vector corresponding to the second attention layer, and connect the first intermediate value and the second intermediate value to obtain a second key vector; Calculate a third intermediate value corresponding to the target feature vector and a fourth intermediate value corresponding to the reference feature vector using a projection matrix of the second value vector corresponding to the second attention layer, and connect the third intermediate value and the fourth intermediate value to obtain a second value vector; The reference object enhancement matrix is ​​calculated according to the second query vector, the second key vector and the second value vector using a softmax function.

5. The object segmentation method of video images according to claim 1, characterized in that: The feature extraction module includes a cascaded feedforward neural network, a normalization layer and a multi-layer perceptron, and the object self-attention matrix, the reference object enhancement matrix and the test reference correlation matrix are input into the feature extraction module to obtain three different output matrices, including: The object self-attention matrix, the reference object enhancement matrix and the test reference correlation matrix are respectively used as intermediate input matrices; Inputting the intermediate input matrix into the feedforward neural network for feature extraction to obtain a first feature; Inputting the first feature into the normalization layer for feature extraction to obtain a second feature; The second feature is input into the multi-layer perceptron for feature extraction to obtain the output matrix.

6. The method for object segmentation of a video image according to claim 2, characterized in that: The determining, based on the time feature database, a target feature value of the object feature matrix under each object feature comprises: Acquire the feature value of each of the object features in the time feature database, and segment each of the historical feature values ​​from the feature value; The similarity between each of the historical feature values ​​and the object feature matrix is ​​calculated, and the target feature value corresponding to the object feature is selected from the historical feature values ​​according to the similarity.

7. The method for object segmentation of a video image according to claim 6, characterized in that: The decoding process of the target feature value to obtain a target mask result includes: Obtaining an overall correlation feature according to the target feature value of each of the object features; The overall associated features are input into a decoder for decoding processing to obtain the target mask result.

8. The method for object segmentation of a video image according to any one of claims 1 to 7, characterized in that: The selecting a reference video frame from all historical video frames before the test video frame comprises: Selecting the first historical video frame as the first reference video frame in time sequence; Based on a preset selection rule, at least one of the other historical video frames is selected as the other reference video frame.

9. A device for segmenting video images, characterized in that: include: The first feature extraction module is used to obtain a test video frame in the target video, select a reference video frame from all historical video frames before the test video frame, and calculate a test feature vector of the test video frame and a reference feature vector of the reference video frame; A second feature extraction module: used to obtain a feature value corresponding to at least one object feature from a time feature database, and perform feature extraction on the feature value to obtain a target feature vector, wherein the feature value is used to indicate a position of the object feature in at least one of the historical video frames; The third feature extraction module is used to perform feature extraction on the target feature vector, the reference feature vector and the test feature vector by using a plurality of cascaded multi-level feature association modules to obtain an object feature matrix. The multi-level feature association module calculates an object self-attention matrix, a reference object enhancement matrix and a test reference correlation matrix according to three different input matrices, and inputs the object self-attention matrix, the reference object enhancement matrix and the test reference correlation matrix into a feature extraction module to obtain three different output matrices. The output matrix corresponding to the test reference correlation matrix of the last multi-level feature association module is the object feature matrix. Target segmentation module: used to determine the target feature value of the object feature matrix under each object feature based on the time feature database, decode the target feature value to obtain a target mask result, and the target mask result includes a segmentation result of each object feature; The process of the multi-level feature association module calculating the object self-attention matrix, the reference object enhancement matrix and the test reference correlation matrix according to three different input matrices comprises the following steps: Calculate the object self-attention matrix based on the first input matrix; Calculate a reference object enhancement matrix based on the first input matrix and the second input matrix; Calculating a test-reference correlation matrix based on the second input matrix and the third input matrix; The three input matrices of the first multi-level feature association module are the target feature vector, the reference feature vector and the test feature vector, respectively, and the three input matrices of other multi-level feature association modules are the three output matrices obtained by the feature extraction module of the previous multi-level feature association module based on the object self-attention matrix, the reference object enhancement matrix and the test reference correlation matrix.

10. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the object segmentation method of video images according to any one of claims 1 to 8 when executing the computer program.

11. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for segmenting an object of a video image according to any one of claims 1 to 8 is implemented.

Images