A video object segmentation system, method, electronic device and medium

Abstract

The application discloses a video target segmentation system, method, electronic equipment and medium, and relates to the field of video processing, which comprises receiving an original video and target annotation information; a feature extraction module comprises a backbone network and a structure feature extraction module, the structure feature extraction module enhances the structure information in the video frame embedded feature in the corresponding stage of the backbone network to obtain an enhanced embedded feature; a memory module stores the segmentation information of the first frame and historical frames; a matching module matches the enhanced embedded feature of the current frame in the original video with the segmentation information of the memory module; and based on the matched segmentation embedded tensor, a video target segmentation result is obtained. Through the enhancement of the video frame embedded feature and the construction of the memory module containing a compressed memory unit, the target structure information is effectively utilized, the storage occupation in the memory module is reduced, and the influence of the point-to-point matching process of the points with similar appearance features is reduced, so that the robustness of the matching process is enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, and in particular to a video target segmentation system and method, an electronic device and a medium. BACKGROUND

[0002] At present, video target segmentation technology can be applied to target tracking, video editing, video conferencing and automatic driving and other computer vision-based products and applications, and its technical essence is to segment the target of interest in a video sequence. The target segmentation result can be a single target or multiple targets, the target to be segmented is called foreground, and other areas are called background. According to the level of supervision required, video target segmentation technology is divided into unsupervised technology, semi-supervised technology and supervised technology. Unsupervised target segmentation (UVOS) technology automatically segments the main target without manual annotation, and usually they assume that the motion of the target to be segmented is different from the surrounding environment. Semi-supervised target segmentation (SVOS) technology requires manual annotation in the first frame to mark the target area of interest, and then uses the manual label to segment and track the target in subsequent video frames. Since only the first frame and its mask are needed as a guide, semi-supervised video target segmentation is also called one-shot video object segmentation (OVOS), and supervised technology is based on a specific scene, which requires manual correction of the segmentation result during the segmentation process.

[0003] Existing semi-supervised video target segmentation technology can be divided into online learning-based target segmentation technology, propagation-based target segmentation technology and matching-based target segmentation technology. Among them, the matching-based video target segmentation technology has the following problems:

[0004] (1) Ignoring the structural features of the target, the existing technology uses a convolutional neural network or an image block-based model or an image block-based multilayer perceptron to extract video frame features, without considering the structural shape features of targets such as rods and ropes;

[0005] (2) The point-to-point matching process is easily disturbed by feature points with similar appearance characteristics, the existing technology performs point-to-point matching between features, which is easily disturbed by some noise points in the video frame to produce false matching;

[0006] (3) The existing technology stores historical frame segmentation information into a memory unit periodically in addition to the first frame and the previous frame to perform the subsequent matching process, however, as the length of the processed video increases, the memory unit will store too much. SUMMARY

[0007] To solve at least one problem mentioned in the background art, the present application provides a video object segmentation system, method, electronic device and medium, which extracts the structural features of each object in the video frame by designing and introducing a minimum spanning tree and a graph convolutional neural network, and then constructs a compressed memory unit composed of prototype features obtained by clustering the first frame features, spatio-temporal features and algorithm to improve the original long-term memory unit, thereby solving the problems of the prior art that the matching process is easily disturbed by similar appearance feature points and the memory unit storage occupies too much space.

[0008] The specific technical solutions provided by the embodiments of the present application are as follows:

[0009] In a first aspect, a video object segmentation system is provided, which comprises:

[0010] A data input module, which receives an original video and target annotation information;

[0011] A feature extraction module, which comprises a backbone network and a structural feature extraction module arranged at different stages of the backbone network, the backbone network extracts video frame embedding features of each frame in the original video, and the structural feature extraction module enhances the structural information in the video frame embedding features at the corresponding stage of the backbone network to obtain enhanced embedding features;

[0012] A memory module, which stores segmentation information of the first frame and historical frames obtained according to the target annotation information;

[0013] A matching module, which matches the enhanced embedding features of the current frame in the original video with the segmentation information of the memory module to obtain a segmentation embedding tensor;

[0014] Based on the segmentation embedding tensor, a video object segmentation result is obtained.

[0015] In a specific embodiment, the system further comprises:

[0016] The backbone network is provided with at least four stages;

[0017] The structural feature extraction module is inserted between the second stage and the third stage of the backbone network.

[0018] In a specific embodiment, the structural feature extraction module comprises a graph structure generation module and a graph feature extraction module:

[0019] The graph structure generation module obtains the video frame embedding features of the current frame at the current stage of the backbone network;

[0020] constructing an initial graph structure, and calculating a minimum spanning tree on the initial graph structure by an algorithm;

[0021] The minimum spanning tree captures target structure shape information in the video frame embedding feature to output an adjacency matrix and an edge weight matrix of graph data containing target structure information.

[0022] The graph feature extraction module performs node feature encoding, node position encoding, and edge weight encoding of the graph data, uses a graph convolutional neural network to extract target structure features of the graph data, and adds the target structure features to the video frame embedding feature to obtain the enhanced embedding feature.

[0023] In one specific embodiment, the memory module includes a long-term memory unit, a short-term memory unit, and a compressed memory unit.

[0024] Obtain the embedding feature and target annotation information of the first frame, and calculate the segmentation information of the first frame based on the embedding feature and target annotation information of the first frame, and store the segmentation information of the first frame in the long-term memory unit.

[0025] Obtain the segmentation information of the previous frame, and store the segmentation information of the previous frame in the short-term memory unit.

[0026] The segmentation information of the first frame, the segmentation information of the periodic frame, the spatiotemporal segmentation information, and the cumulative prototype feature constitute the compressed memory unit.

[0027] In one specific embodiment, the matching module includes a global matching module, a local matching module, and a compressed matching module.

[0028] Match the enhanced embedding feature of the current frame with the segmentation information in the long-term memory unit to obtain a global segmentation embedding tensor of the current frame.

[0029] Match the enhanced embedding feature of the current frame with the segmentation information in the short-term memory unit to obtain a local segmentation embedding tensor of the current frame.

[0030] Cluster, propagate, and reconstruct the enhanced embedding feature of the current frame with the segmentation information in the compressed memory unit to obtain a reconstructed segmentation tensor of the current frame.

[0031] Combine the global segmentation embedding tensor of the current frame, the local segmentation embedding tensor of the current frame, and the reconstructed segmentation tensor of the current frame to obtain the segmentation embedding tensor.

[0032] In one specific embodiment, the enhanced embedding feature of the current frame is clustered, propagated, and reconstructed with the segmentation information in the compressed memory unit to obtain a reconstructed segmentation tensor of the current frame, specifically including:

[0033] The compressed memory unit comprises: segmentation information of a first frame, segmentation information of a periodic frame, spatiotemporal segmentation information, and accumulated prototype features;

[0034] The clustering operation in the compressed matching module is performed on the segmentation information of the first frame, the segmentation information of the periodic frame, the spatiotemporal segmentation information, and the accumulated prototype features, to obtain prototype features and a prototype embedding tensor;

[0035] The propagation operation and the reconstruction operation in the compressed matching module are performed on the enhanced embedding features of the current frame, the prototype features, and the prototype embedding tensor, to obtain a reconstructed segmentation tensor of the current frame.

[0036] In one specific embodiment, the system further comprises:

[0037] A feature decoding module that performs feature decoding on the segmentation embedding tensor to obtain a video target segmentation result;

[0038] A result output module that stores or transmits the video target segmentation result.

[0039] In a second aspect, a video target segmentation method is provided, which comprises:

[0040] Receiving an original video and target annotation information;

[0041] Extracting video frame embedding features of each frame in the original video, and enhancing the video frame embedding features at corresponding stages in a backbone network to obtain enhanced embedding features;

[0042] Obtaining enhanced embedding features of a current frame in the original video, matching the enhanced embedding features of the current frame with segmentation information in a memory module to obtain a segmentation embedding tensor;

[0043] Performing feature decoding on the segmentation embedding tensor to obtain the video target segmentation result.

[0044] In a third aspect, an electronic device is provided, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the following steps when executing the computer program:

[0045] Step A: receiving an original video and target annotation information;

[0046] Step B: extracting video frame embedding features of each frame in the original video, and enhancing the video frame embedding features at corresponding stages in a backbone network to obtain enhanced embedding features;

[0047] Step C: Obtain the enhanced embedding feature of the current frame in the original video, match the enhanced embedding feature of the current frame with the segmentation information in the memory module, and obtain a segmentation embedding tensor;

[0048] Step D: Perform feature decoding on the segmentation embedding tensor to obtain the video target segmentation result.

[0049] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. When the computer program is executed by a processor, the following steps are implemented:

[0050] Step A: Receive an original video and target annotation information;

[0051] Step B: Extract a video frame embedding feature of each frame in the original video, and enhance the video frame embedding feature of a corresponding stage in a backbone network to obtain an enhanced embedding feature;

[0052] Step C: Obtain the enhanced embedding feature of the current frame in the original video, match the enhanced embedding feature of the current frame with the segmentation information in the memory module, and obtain a segmentation embedding tensor;

[0053] Step D: Perform feature decoding on the segmentation embedding tensor to obtain the video target segmentation result.

[0054] The embodiments of the present application have the following beneficial effects:

[0055] 1. The data input module, feature extraction module, memory module and matching module provided by the embodiment of the application obtain a segmentation embedding tensor, and a video target segmentation result is obtained based on the segmentation embedding tensor; specifically, the data input module is configured to receive an original video and target annotation information and unify the format of the original video, and store a video target annotation file given by a first frame of the original video, which is usually in a single channel format; the feature extraction module includes a backbone network and a structure feature extraction module, the backbone network extracts video frame embedding features of each frame in the original video, and the structure information in the video frame embedding features at a corresponding stage in the backbone network is enhanced by the structure feature extraction module, so that enhanced embedding features of each frame are obtained; since the backbone network generally has multiple stages, the structure feature extraction module is inserted into different stages of the backbone network, and the structure information in the video frame features at different stages in the backbone network is enhanced by the structure feature extraction module, so that enhanced embedding features of each frame are obtained; the segmentation information of the first frame and historical frames is stored in the memory module, the matching module obtains enhanced embedding features of a current frame in the original video, matches the enhanced embedding features of the current frame with the segmentation information of the memory module, and obtains a segmentation embedding tensor; the segmentation embedding tensor is subjected to feature decoding, and a final video target segmentation result is obtained; in the application, the video frame embedding features are enhanced, and a memory module containing a compression memory unit is constructed, so that the target structure information is effectively utilized, the storage occupation in the memory module is reduced, and the influence of the point-to-point matching process of the points with similar appearance features is reduced, thereby enhancing the robustness of the matching process. BRIEF DESCRIPTION OF DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative effort.

[0057] Figure 1 A schematic diagram of a video segmentation system according to the application is shown;

[0058] Figure 2 A structural schematic diagram of a feature extraction module according to the application is shown;

[0059] Figure 3 A structural schematic diagram of a memory module and a matching module according to the application is shown;

[0060] Figure 4 A structural schematic diagram of a feature decoding module according to the application is shown;

[0061] Figure 5A schematic diagram showing an example picture output by a picture structure generation module according to the present application;

[0062] Figure 6 A schematic diagram showing a result segmented by a video current segmentation system according to the present application;

[0063] Figure 7 A schematic diagram showing a video object segmentation method according to the present application;

[0064] Figure 8 A schematic diagram showing an electronic device according to the present application. DETAILED DESCRIPTION

[0065] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0066] As described in the background, the video object segmentation technology based on online learning uses the first frame and the given object mask to fine-tune the network to learn the appearance features of the video object, so as to realize the segmentation of the video object. On this basis, the video object segmentation technology based on meta-learning has developed, which also belongs to the paradigm of online learning. Different from the previous object segmentation technology based on online learning, it uses the first frame and the given object mask and the historical frame and the segmentation result to construct the network layer supporting the fine-tuning part instead of all layers. In this way, the time performance of the object segmentation technology based on online learning is improved. The object segmentation technology based on propagation transmits the segmentation result of the previous frame to the current frame by utilizing the correspondence between the video frames. Some such technologies propagate the segmentation results of the first frame and the historical frame in addition to the first frame to make more full use of the historical segmentation information. The video object tracking method is also introduced into such technologies to realize the area propagation from the previous frame to the current frame. The video object segmentation technology based on matching has the problems of ignoring the structural features of the object, being easily disturbed by the points with similar appearance features in the point-to-point matching process, and the large storage data volume of the memory unit. The present application proposes a video object segmentation system, method, electronic device and medium.

[0067] Embodiment one

[0068] A video object segmentation system, as shown in Figure 1 The system comprises:

[0069] (1) a data input module, which receives the original video and the object annotation information.

[0070] Specifically, the data input module adjusts the incoming video data to be processed and the corresponding target annotation information into a unified format, such as file naming and storage. Each video has a corresponding annotation folder that stores a video target annotation file given by the first frame of the video, usually in single-channel format. Moreover, the video target segmentation technology proposed by the present application supports giving new target annotations in the middle frames of the video in addition to the annotations of the first frame of the input video.

[0071] (2) A feature extraction module, the feature extraction module comprising a backbone network and feature extraction modules arranged in different stages of the backbone network, the backbone network extracting video frame embedding features of each frame in the original video, and the feature extraction modules enhancing the video frame embedding features of the corresponding stages in the backbone network to obtain enhanced embedding features of each frame.

[0072] Specifically, the feature extraction module is composed of a backbone network and two video target structure feature extraction modules. The backbone network refers to a general computer vision backbone network such as ResNet-50, ResNet-101, and Swin Transformer. Existing video target segmentation technologies all use general backbone networks to extract embedding features of video frames, but these general backbone networks still have deficiencies, so many modules for enhancing backbone networks have been proposed. Specifically, a general computer vision backbone network is generally divided into four stages, i.e., a first stage, a second stage, a third stage, and a fourth stage. Considering the network depth and feature resolution of each stage, placing video target structure feature extraction modules in the second and third stages can avoid excessive computational load and efficiently utilize information at different stages of the backbone network. That is, video target structure feature extraction modules are inserted in parallel in the second and third stages of the backbone network to extract embedding features of video frames at the current stage and thereby enhance the original embedding features. The video target structure feature extraction module is composed of a graph structure generation module and a graph feature extraction module. The embedding features are transformed from two dimensions to one dimension for feature matching after dimension transformation.

[0073] In one specific embodiment, the feature extraction module further comprises a graph structure generation module, which is connected to and exists in pairs with the graph feature extraction module. The graph structure generation module obtains the video frame embedding features of the current frame at the current stage of the backbone network; constructs an initial graph structure, and calculates a minimum spanning tree on the initial graph structure through an algorithm; the minimum spanning tree captures the target structure shape information in the video frame embedding features to output an adjacency matrix and an edge weight matrix of a graph data structure containing target structure information.

[0074] Specifically, as Figure 2As shown, by inputting the video frame embedding features of the current stage of the backbone network into the graph structure generation module, an initial graph structure is constructed. For each feature point at a position, its eight neighborhood feature points, i.e., the upper, lower, left, right, top-left, bottom-left, top-right and bottom-right neighborhood feature points, are connected. The edge weight is calculated using the negative square distance. Then, the Boruvka algorithm is used to obtain the minimum spanning tree on the initial graph structure. The minimum spanning tree connects the feature points with similar features, thereby capturing the shape information of the target structure. An adaptive threshold method is used to disconnect some edges with large similarity differences, thereby obtaining a series of sub-minimum spanning trees, so that regions with different properties are separated. In actual use, in order to avoid falling into a local extremum, a random spanning tree is used to generate a graph structure during network training.

[0075] Further, the graph feature extraction module encodes the node features, node positions and edge weights of the graph data, uses a graph convolutional neural network to extract the target structure features of the graph data, and adds the target structure features to the video frame embedding features to obtain the enhanced embedding features.

[0076] Specifically, the output of the graph structure generation module includes the adjacency matrix and the edge weight matrix of the graph data of the target structure features. The video frame embedding features of the current stage extracted by the backbone network can obtain the node features of the graph data. First, the node features, node positions and edge weights of the graph data are encoded, respectively. Then, the graph convolutional neural network is used to complete the message passing on the graph structure, thereby introducing the target structure features into the input video frame embedding features, so as to obtain the enhanced embedding features. Specifically, the node feature encoding and the edge weight encoding are completed by a multilayer perceptron. The node position encoding can be completed by a SignNet and a multilayer perceptron, or can be completed by a graph Laplacian matrix, or can be completed by a random walk method. The present application uses a GINE graph convolutional neural network. Other graph convolutional networks supporting edge feature input can also be used, without specific limitation.

[0077] (3) a memory module, wherein the memory module stores segmentation information of the first frame and historical frames obtained according to the target annotation information.

[0078] In a specific embodiment, as shown in Figure 3 The memory module is composed of a long-term memory unit, a short-term memory unit and a compressed memory unit, and is used to store the segmentation information of the first frame and historical frames. The embedding features of the first frame and the target annotation information are obtained, and the segmentation information of the first frame is calculated by combining the embedding features of the first frame and the target annotation information, and the segmentation information of the first frame is stored in the long-term memory unit. The segmentation information of the previous frame is obtained, and the segmentation information of the previous frame is stored in the short-term memory unit. The segmentation information of the first frame, the segmentation information of the periodic frame, the spatiotemporal segmentation information and the cumulative prototype feature constitute the compressed memory unit.

[0079] Specifically, the segmentation information of each frame in the long-term memory unit is composed of pairs of key-value (K-V), called segmentation feature-segmentation embedding tensor, which is obtained by applying self-attention in the Transformer to the embedding features of each frame. K is used for similarity calculation between features, and V retains the complete information of the current frame features and is updated according to the similarity value between K. K also performs position embedding to introduce position information in the two-dimensional space of the image, and V performs identity embedding to simultaneously match multiple targets. Among them, the identity embedding mechanism adopts the identity embedding method summarized by AOT technology, that is, identity embedding is realized by constructing an identity matrix and a permutation matrix, each row of the identity matrix is an identity vector corresponding to the identity of a target, and the column is the dimension of the identity vector. Through the identity matrix and the permutation matrix, the target annotation information is converted to the same dimension as V and added to V to realize identity embedding.

[0080] The long-term memory unit stores the segmentation information of the first frame, which is calculated from the embedding features of the first frame of the video and the target annotation information. If there are other frames providing target annotation information in addition to the first frame, the long-term memory unit updates when processing the annotation frame, that is, the segmentation information of the new annotation frame is concatenated with the original segmentation information. The short-term memory unit stores the segmentation information of the last frame, and updates the short-term memory unit according to the K value and the corresponding segmentation result of the frame after processing each frame.

[0081] The compressed memory unit specifically includes segmentation information of the first frame, segmentation information of the periodic frame, spatiotemporal segmentation information, and accumulated prototype features; the segmentation information of the first frame is the segmentation information stored by the long-term memory unit. The segmentation information of the periodic frame is updated at a certain period, that is, the short-term memory information is replaced with the segmentation information of the original periodic frame every 5 frames. Similar to the segmentation information, the prototype features and the corresponding prototype embedding tensor are a pair of key-value (K-V), except that the number of feature points is different. The number of feature points in the segmentation information is HxW, and H and W correspond to the height and width of the current features in the two-dimensional space. The number of feature points in the prototype features and the corresponding prototype value tensor is the set clustering number, which is less than HxW. The propagation operation and the reconstruction operation in the feature matching compression matching submodule are performed on the current frame features, the prototype features, and the corresponding prototype embedding tensor to obtain the segmentation information of the current frame after compression matching, that is, the reconstructed segmentation features and the reconstructed segmentation embedding tensor. The new prototype features and the accumulated prototype features are added in a certain proportion to update the accumulated prototype features for use when processing the next frame. The feature of the first P feature points with high posterior probability value of the current frame segmentation feature belonging to the prototype feature in the propagation operation in the feature matching compression matching submodule is reserved. After processing the current frame, the segmentation embedding tensor corresponding to the segmentation features of the frame can be obtained by embedding the segmentation result of the frame. If the frame is a non-periodic frame, the selected first P feature point vector of the frame and the corresponding feature embedding vector are stored in the spatiotemporal segmentation information, wherein the segmentation embedding vector corresponding to the first P feature points is calculated by the similarity weight of the first P feature points and all feature points in the frame segmentation features, and the weighted aggregation of the frame segmentation embedding tensor. The number of feature point vectors and corresponding segmentation embedding vectors stored in the spatiotemporal segmentation information is L. The spatiotemporal segmentation information supports processing of a longer video, but as the length of the processed video increases, the number of feature point vectors and corresponding segmentation embedding vectors stored in the spatiotemporal segmentation information also increases. When L is reached, P non-frequent feature point vectors and corresponding segmentation embedding vectors are removed from the first T feature points in the spatiotemporal segmentation information to store new feature point vectors and corresponding segmentation embedding vectors. The non-frequent degree of the feature points is obtained according to the similarity weight value used when selecting the feature points.

[0082] (4) matching module

[0083] The matching module obtains the enhanced embedding features of the current frame in the original video, matches the enhanced embedding features of the current frame with the segmentation information of the memory module, and obtains the segmentation embedding tensor.

[0084] In a specific embodiment, the matching module includes a global matching module, a local matching module, and a compressed matching module; matching the enhanced embedding features of the current frame with the segmentation information in the long-term memory unit to obtain the global segmentation embedding tensor of the current frame; matching the enhanced embedding features of the current frame with the segmentation information in the short-term memory unit to obtain the local segmentation embedding tensor of the current frame; clustering, propagating, and reconstructing the enhanced embedding features of the current frame with the segmentation information in the compressed memory unit to obtain the reconstructed segmentation tensor of the current frame; and combining the global segmentation embedding tensor, the local segmentation embedding tensor, and the reconstructed segmentation tensor of the current frame to obtain the segmentation embedding tensor.

[0085] Specifically, such as Figure 3 As shown, global matching represents the matching process between the segmentation features of the current frame and the segmentation information in the long-term memory unit; local matching represents the matching process between the segmentation features of the current frame and the segmentation information in the short-term memory unit; and compressed matching represents the matching process between the segmentation features of the current frame and the compressed memory unit. The feature matching module is obtained by inserting the designed global matching, local matching, and compressed matching sub-modules into the Transformer module containing multi-head self-attention. Specifically, the enhanced embedding features of the input current frame are first processed by the multi-head self-attention layer and then subjected to linear transformation to obtain the segmentation features. The segmentation features of the current frame are then globally matched with the segmentation information in the long-term memory unit to obtain the global segmentation embedding tensor of the current frame. This operation is completed using multi-head cross-attention and the linear transformation is not repeated to transform the segmentation features and segmentation embedding tensor.

[0086] The segmentation features of the current frame are locally matched with the segmentation information in the short-term memory unit to obtain the local segmentation embedding tensor of the current frame. This operation is also completed using multi-head cross-attention. Unlike global matching, local matching restricts the matching range between feature points in the current segmentation features and feature points in the segmentation information in the short-term memory unit. A local window of the same size centered on the feature point is set for each feature point in the current segmentation features. Each feature point can only be matched with feature points in the segmentation information in the short-term memory unit that are located in the local window.

[0087] The compressed matching process involves clustering, propagating, and reconstructing the enhanced embedding features of the current frame with the segmentation information in the compressed memory unit to obtain the reconstructed segmentation tensor of the current frame. Specifically, the compressed memory unit includes: segmentation information of the first frame, segmentation information of the periodic frames, spatiotemporal segmentation information, and accumulated prototype features; performing clustering operations in the compressed matching module on the segmentation information of the first frame, the segmentation information of the periodic frames, the spatiotemporal segmentation information, and the accumulated prototype features to obtain prototype features and prototype embedding tensors; and performing propagation and reconstruction operations in the compressed matching module on the enhanced embedding features, prototype features, and prototype embedding tensors of the current frame to obtain the reconstructed segmentation tensor of the current frame.

[0088] Specifically, compressed matching includes three operations: clustering, propagation, and reconstruction. First, the segmentation information of the first frame, the segmentation information of the periodic frames, and the spatiotemporal segmentation information in the compressed memory unit are concatenated and then clustered together with the accumulated prototype features to obtain new prototype features and corresponding prototype embedding tensors. The clustering operation finds the class centers of the concatenated segmentation information, i.e., the prototype features and their corresponding prototype embedding tensors. A Gaussian Mixture Model (GMM) is used to model the feature distribution in the segmentation information; the number of clusters is the number of Gaussian mixtures. The cluster centers, i.e., the prototype features, are obtained iteratively through the EM algorithm. The input accumulated prototype features are the initial cluster centers. A propagation operation is performed on the current frame segmentation features and the prototype features obtained by the EM algorithm to obtain the posterior probability matrix of the current frame segmentation features belonging to each cluster center in the prototype features. Reconstruction operations are performed on the prototype features and their corresponding prototype embedding tensors to obtain the reconstructed segmentation features and reconstructed segmentation embedding tensors for the current frame. The three operations—clustering, propagation, and reconstruction—mainly include matrix multiplication and normalization operations, where matrix multiplication is implemented using Einstein's summation formula.

[0089] Clustering operations model the feature distribution in segmentation information using Gaussian Mixture Models (GMM), as shown in equations (1) and (2):

[0090]

[0091] Where, k m This represents the segmentation feature after the first frame segmentation feature, periodic frame segmentation feature, and spatiotemporal segmentation feature are concatenated in the compressed memory unit. The total number of feature points it contains is B, where b = 1, 2, ..., B, and b is the feature point index number. The segmentation feature k of the compressed memory unit represents the segmentation feature. m The vector of the b-th feature point The probability; j = 1, 2, ..., N, where j represents the index number of the Gaussian model; Represents feature point vector The probability value in the j-th Gaussian model; z is the membership degree latent variable.

[0092]

[0093] Where, k m This represents the segmentation feature after the first frame segmentation feature, periodic frame segmentation feature, and spatiotemporal segmentation feature are concatenated in the compressed memory unit. The total number of feature points it contains is B, where b = 1, 2, ..., B, and b is the feature point index number. Represents feature point vector The probability value in the j-th Gaussian model, where z is the membership degree latent variable.

[0094] Formula (3) further gives The calculation method and formula are as follows:

[0095]

[0096] Where E is the dimension of each feature point vector in the segmentation feature; σ is the variance of the Gaussian model; is the class center of the j-th Gaussian model; N is the number of Gaussian mixtures.

[0097] The posterior probability of the membership latent variable z can be obtained according to Bayes' rule, and soft clustering of the segmentation features can be achieved as shown in formula (3). Represents the vector of the b-th feature point The posterior probability of belonging to the j-th Gaussian model. p(z=j) represents the probability of the j-th Gaussian model appearing among all Gaussian models, and is uniformly distributed. j, l = 1, 2, ..., N are the index numbers of the Gaussian models, and N is the number of Gaussian models. Represents feature point vector The probability value in the j-th Gaussian model. This is determined by using negative clustering distance. and temperature coefficient 2σ 2 Formula (3) can be implemented using the Softmax operation. The N cluster centers, i.e., all prototype features k, are obtained through iterative EM algorithm. o The cumulative prototype features of the input are used as the initial cluster centers.

[0098] The posterior probability of z obtained using formula (3) The prototype embedding tensor v corresponding to the concatenated segmented embedding tensor of the first frame, the periodic frame, and the spatiotemporal segmented embedding tensor in the compressed memory unit is calculated. o The prototype embedding tensor of the j-th feature point vector The calculation method of is shown in formula (4), j=1, 2..., N is the index number of the prototype feature, and N is the number of feature points contained in the prototype feature, which also corresponds to the number of Gaussian models and clusters. b=1, 2..., B is the index number of the feature point, and B is the segmented embedded tensor v after cascading m The total number of feature points contained. As described previously k m and v m respectively represent the segmented features and the segmented embedded tensor after cascading in the compressed memory unit, which form a pair of key-value (K-V) and are referred to as segmentation information. is the bth feature point vector in the segmented feature, is the bth feature point vector in the segmented embedded tensor. represents the bth feature point belongs to the jth Gaussian model. The jth feature point vector of the prototype embedded tensor is obtained by weighting all the feature point vectors in the segmented embedded tensor, The posterior probability is corresponding weight.

[0099]

[0100] wherein j is the index number of the prototype feature; N is the number of feature points contained in the prototype feature; b is the index number of the feature point; k m is the segmented feature after cascading in the compressed memory unit; v m is the segmented embedded tensor after cascading in the compressed memory unit; is the bth feature point vector in the segmented feature, is the bth feature point vector in the segmented embedded tensor; represents the bth feature point belongs to the jth Gaussian model.

[0101] The posterior probability matrix of the segmented feature k c of the current frame to each cluster center in the prototype feature k o is obtained by performing a propagation operation i=1, 2..., HxW is the index number of the feature point in the segmented feature of the current frame, H and W correspond to the height and width of the current feature in the two-dimensional space, and j=1, 2..., N is the index number of the prototype feature.

[0102] The posterior probability matrix is combined with the prototype feature k o and the corresponding prototype embedded tensor v o respectively to perform a reconstruction operation to obtain the reconstructed segmented feature and the reconstructed segmented embedded tensor v of the current framer Reconstructing the segmented embedding tensor v r The vector of the i-th feature point The reconstruction process is shown in formula (5). This can be further expanded to the rightmost part of formula (5). The calculation method is the same as that for the numerator in formula (3), that is... for The probability value in the j-th Gaussian model. Let i be the vector of the i-th feature point in the segmentation features of the current frame. Let be the vector of the j-th feature point in the prototype features, and be the class center of the j-th Gaussian model. The three operations of clustering, propagation, and reconstruction mainly include matrix multiplication and normalization operations, where matrix multiplication is implemented using Einstein's summation formula.

[0103]

[0104] in, for The probability value in the j-th Gaussian model; This is the vector of the i-th feature point in the segmentation features of the current frame; is the vector of the j-th feature point in the prototype features.

[0105] The reconstructed segmentation embedding tensor of the current frame obtained through the compression matching module is summed with the global embedding tensor and the local embedding tensor, and then passed through feedforward layers and activation layers to obtain the final segmentation embedding tensor. To improve network performance, three stacked feature matching modules are used in the feature matching process, each with its corresponding memory unit.

[0106] Based on the segmentation embedding tensor, a video target segmentation result is obtained. In a specific embodiment, the system further includes a feature decoding module and a result output module. The feature decoding module performs feature decoding on the segmentation embedding tensor to obtain the video target segmentation result; the result output module stores or transmits the video target segmentation result. Specifically, the result output module is responsible for storing the video target segmentation result obtained by the segmentation technique to a hard disk or transmitting it out, and the video target segmentation result of each frame is in single-channel format.

[0107] In one specific embodiment, the segmentation embedding tensor obtained by the feature matching module undergoes a dimensionality transformation from one-dimensional to two-dimensional to obtain the matched features. Simultaneously, multi-scale features of the backbone network are input to the feature decoding module via skip connections. For example... Figure 4As shown, a feature pyramid-based (FPN) convolutional decoding network is then used to gradually increase the resolution while simultaneously reducing the number of channels, starting from 1 / 16 of the video frame image resolution, then increasing to 1 / 8, and finally 1 / 4 of the resolution. After one convolutional layer, the target identity information in the segmentation result is recovered through the permutation matrix in the identity embedding mechanism of AOT technology. Finally, the image is upsampled to the resolution of the original video frame image, and then subjected to a softmax operation, with the target number with the highest probability being taken to obtain the final video target segmentation result.

[0108] In a specific embodiment, in addition to the main modules of the video object segmentation technology and system with enhanced structural features mentioned above, the deep neural network in the entire system can be trained end-to-end. Given a video dataset, a preset number of video frames are taken for each video sequence at random intervals for training. Furthermore, video sequences can be synthesized using semantic segmentation datasets and saliency detection datasets through data augmentation techniques. Using the synthesized video sequences to pre-train the network, followed by training on real video datasets, can further improve system performance.

[0109] like Figure 5 The diagram shows the results output by the graph structure generation module in the segmentation scheme of this embodiment, with some video frames selected from three video sequences. For ease of illustration, the graph structure in some small regions of the diagram is omitted. These results show that the output graph structure is connected within the target and background regions, corresponding to different sub-minimum spanning trees, but disconnected at the boundaries of different regions, separating the different sub-minimum spanning trees. Examples include the boundary between the camel and the background region in the first video, the boundary between the four-wheeled vehicle and the background region in the third video, and the boundary between the person carrying things and the background region in the fourth video. The obtained graph structure can be used to simultaneously enhance the features of similar appearance regions in the video.

[0110] like Figure 6 The image shows the processing results of the video sequence segmented using the segmentation method in this embodiment. The left side shows the first frame of the video and its target annotation information, while the right side shows the segmentation results of subsequent video frames. It can be seen that the video segmentation technology proposed in this invention can distinguish similar targets without being affected by the second camel appearing later in the first video; it can effectively handle scenes of human interaction, such as the person and the four-wheeled vehicle in the third video, and the person and the box being transported in the fourth video; it can accurately segment the rope used for surfing in the second video, and the branch-like targets such as the four-wheeled vehicle's handrail in the fourth video; and it can handle complex backgrounds, such as the relatively complex background interference in the fourth video.

[0111] By the system in the embodiment, the minimum spanning tree is used to generate the graph structure of the video frame features, in order to avoid falling into a local extremum, a random spanning tree is used to generate the graph structure in the network training stage, and on the basis of the generated graph structure, the graph convolutional neural network is used to extract and enhance the structural features of the video frame features; further, the prototype features of the first frame, the previous frame and the historical frame are obtained by clustering based on an EM algorithm, the prototype features are used for compression matching with the current frame features to enhance the original matching process, the corresponding compression memory unit is updated after processing the current video frame each time, and the memory module containing the compression memory unit is constructed, so that the storage occupation in the memory module is reduced, and the influence of the point-to-point matching process of the feature points with similar appearances is reduced, thereby enhancing the robustness of the matching process.

[0112] Embodiment two

[0113] Corresponding to the above embodiment, the application provides a video target segmentation method, as shown in Figure 7 The method comprises the following steps:

[0114] Step S1: receiving an original video and target annotation information.

[0115] The incoming video data and target annotation information to be processed are adjusted to a unified format by a data input module, such as file naming and storage, and each video has a corresponding annotation folder, which stores the video target annotation file given by the first frame of the video, usually in a single channel format. Moreover, the video target segmentation technology provided by the application supports giving new target annotations in the middle frames of the video in addition to the annotations of the first frame of the input video.

[0116] Step S2: extracting video frame embedding features of each frame in the original video, and enhancing structural information in the video frame embedding features corresponding to the stage in the backbone network to obtain enhanced embedding features of each frame.

[0117] Specifically, the backbone network is provided with at least four stages; the feature extraction module is inserted in the second stage and the third stage in the backbone network. The structural feature extraction module comprises a graph structure generation module and a graph feature extraction module. Specifically, the following steps are included:

[0118] Step 2.1: the graph structure generation module obtains the video frame embedding features of the current frame of the current stage of the backbone network.

[0119] Step 2.2: constructing an initial graph structure, and calculating a minimum spanning tree on the initial graph structure by an algorithm;

[0120] Step 2.3: The minimum spanning tree captures the target structure shape information in the video frame embedding features to output an adjacency matrix and an edge weight matrix of the graph data containing the target structure information;

[0121] Step 2.4: A graph feature extraction module, which performs node feature encoding, node position encoding, and edge weight encoding of the graph data, uses a graph convolutional neural network to extract target structure features of the graph data, and adds the target structure features to the video frame embedding features to obtain the enhanced embedding features.

[0122] Step S3: Obtain the enhanced embedding features of the current frame in the original video, match the enhanced embedding features of the current frame with the segmentation information in the memory module, and obtain the segmentation embedding tensor.

[0123] Specifically, the memory module includes a long-term memory unit, a short-term memory unit, and a compressed memory unit.

[0124] Obtain the embedding features of the first frame and the target annotation information, and calculate the segmentation information of the first frame in combination with the embedding features of the first frame and the target annotation information, and store the segmentation information of the first frame in the long-term memory unit; obtain the segmentation information of the previous frame, and store the segmentation information of the previous frame in the short-term memory unit; obtain the segmentation information of the first frame, the segmentation information of the periodic frame, the spatiotemporal segmentation information, and the cumulative prototype feature to form the compressed memory unit.

[0125] In one specific embodiment, the matching module includes a global matching module, a local matching module, and a compressed matching module; match the enhanced embedding features of the current frame with the segmentation information in the long-term memory unit to obtain the global segmentation embedding tensor of the current frame; match the enhanced embedding features of the current frame with the segmentation information in the short-term memory unit to obtain the local segmentation embedding tensor of the current frame; cluster, propagate, and reconstruct the enhanced embedding features of the current frame with the segmentation information in the compressed memory unit to obtain the reconstructed segmentation tensor of the current frame; and combine the global segmentation embedding tensor of the current frame, the local segmentation embedding tensor of the current frame, and the reconstructed segmentation tensor of the current frame to obtain the segmentation embedding tensor.

[0126] In one specific embodiment, the enhanced embedding features of the current frame are clustered, propagated, and reconstructed with the segmentation information in the compressed memory unit to obtain the reconstructed segmentation tensor of the current frame, specifically including:

[0127] The compression memory unit comprises: segmentation information of a first frame, segmentation information of a periodic frame, spatiotemporal segmentation information, and accumulated prototype features; a clustering operation in the compression matching module is performed on the segmentation information of the first frame, the segmentation information of the periodic frame, the spatiotemporal segmentation information, and the accumulated prototype features to obtain prototype features and a prototype embedding tensor; a propagation operation and a reconstruction operation in the compression matching module are performed on the enhanced embedding features of the current frame, the prototype features, and the prototype embedding tensor to obtain a reconstructed segmentation tensor of the current frame.

[0128] Step S4: performing feature decoding on the segmentation embedding tensor to obtain the video target segmentation result.

[0129] Specifically, the feature decoding module is used to perform feature decoding on the segmentation embedding tensor to obtain the video target segmentation result, and the result output module is used to store or transmit the video target segmentation result.

[0130] Embodiment Three

[0131] An electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the following steps when executing the computer program:

[0132] Step 101: receiving an original video and target annotation information.

[0133] Step 102: extracting video frame embedding features of each frame in the original video, and enhancing the video frame embedding features at corresponding stages in the backbone network to obtain enhanced embedding features;

[0134] Step 103: obtaining enhanced embedding features of a current frame in the original video, matching the enhanced embedding features of the current frame with segmentation information in a memory module to obtain a segmentation embedding tensor;

[0135] Step 104: performing feature decoding on the segmentation embedding tensor to obtain the video target segmentation result. In a specific embodiment, the backbone network in step 102 is provided with at least four stages; the feature extraction module is inserted between the second stage and the third stage of the backbone network. The structural feature extraction module includes a graph structure generation module and a graph feature extraction module: the graph structure generation module obtains the video frame embedding features of the current frame of the current stage of the backbone network; an initial graph structure is constructed, and a minimum spanning tree on the initial graph structure is calculated by an algorithm; the minimum spanning tree captures the target structure shape information in the video frame embedding features to output an adjacency matrix and an edge weight matrix containing the graph data of the target structure information; the graph feature extraction module performs node feature coding, node position coding, and edge weight coding on the graph data, uses a graph convolutional neural network to extract the target structure feature of the graph data, and adds the video frame embedding features to obtain the enhanced embedding features.

[0136] In a specific embodiment, the memory module in step 103 includes a long-term memory unit, a short-term memory unit, and a compressed memory unit.

[0137] The embedding features and target annotation information of the first frame are obtained, and the segmentation information of the first frame is calculated based on the embedding features and target annotation information of the first frame, and the segmentation information of the first frame is stored in the long-term memory unit; the segmentation information of the previous frame is obtained and stored in the short-term memory unit; the segmentation information of the first frame, the segmentation information of the periodic frame, the spatiotemporal segmentation information, and the cumulative prototype feature form the compressed memory unit.

[0138] In a specific embodiment, the matching module includes a global matching module, a local matching module, and a compressed matching module; the enhanced embedding features of the current frame are matched with the segmentation information in the long-term memory unit to obtain a global segmentation embedding tensor of the current frame; the enhanced embedding features of the current frame are matched with the segmentation information in the short-term memory unit to obtain a local segmentation embedding tensor of the current frame; the enhanced embedding features of the current frame are clustered, propagated, and reconstructed with the segmentation information in the compressed memory unit to obtain a reconstructed segmentation tensor of the current frame; the global segmentation embedding tensor of the current frame, the local segmentation embedding tensor of the current frame, and the reconstructed segmentation tensor of the current frame are combined to obtain the segmentation embedding tensor.

[0139] In a specific embodiment, the enhanced embedding features of the current frame are clustered, propagated, and reconstructed with the segmentation information in the compressed memory unit to obtain a reconstructed segmentation tensor of the current frame, specifically including:

[0140] The compressed memory unit includes segmentation information of a first frame, segmentation information of a periodic frame, spatiotemporal segmentation information, and accumulated prototype features; clustering operations in the compressed matching module are performed on the segmentation information of the first frame, the segmentation information of the periodic frame, the spatiotemporal segmentation information, and the accumulated prototype features to obtain prototype features and a prototype embedding tensor; and propagation operations and reconstruction operations in the compressed matching module are performed on the enhanced embedding features of the current frame, the prototype features, and the prototype embedding tensor to obtain a reconstructed segmentation tensor of the current frame.

[0141] In one specific embodiment, step 104 specifically includes performing feature decoding on the segmentation embedding tensor by a feature decoding module to obtain a video target segmentation result, and a result output module stores or transmits the video target segmentation result.

[0142] In one embodiment, an electronic device, which can be a server, has an internal structure diagram as shown in Figure 8 The electronic device includes a processor, a memory, a network interface, and a database connected through a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the electronic device is used to store raw video data. The network interface of the electronic device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement a video target segmentation method.

[0143] Those skilled in the art can understand that Figure 8 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0144] Embodiment Four

[0145] In one embodiment of the present application, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the following steps:

[0146] Step 201: receiving raw video and target annotation information.

[0147] Step 202: extracting video frame embedding features of each frame in the raw video, and enhancing the video frame embedding features at the corresponding stage in the backbone network to obtain enhanced embedding features;

[0148] Step 203: Obtain the enhanced embedding feature of the current frame in the original video, match the enhanced embedding feature of the current frame with the segmentation information in the memory module, and obtain the segmentation embedding tensor;

[0149] Step 204: Perform feature decoding on the segmentation embedding tensor to obtain the video target segmentation result.

[0150] In one specific embodiment, the backbone network in step 202 is provided with at least four stages; the feature extraction module is inserted between the second stage and the third stage in the backbone network. The structural feature extraction module includes a graph structure generation module and a graph feature extraction module: the graph structure generation module obtains the video frame embedding feature of the current frame of the current stage of the backbone network; an initial graph structure is constructed, and a minimum spanning tree on the initial graph structure is calculated by an algorithm; the minimum spanning tree captures the target structure shape information in the video frame embedding feature, to output an adjacency matrix and an edge weight matrix containing the graph data of the target structure information; the graph feature extraction module performs node feature coding, node position coding, and edge weight coding on the graph data, uses a graph convolutional neural network to extract the target structure feature of the graph data, and adds the video frame embedding feature to obtain the enhanced embedding feature.

[0151] In one specific embodiment, in step 203, the memory module includes a long-term memory unit, a short-term memory unit, and a compressed memory unit;

[0152] The embedding feature of the first frame and the target annotation information are obtained, and the segmentation information of the first frame is calculated by combining the embedding feature of the first frame and the target annotation information, and the segmentation information of the first frame is stored in the long-term memory unit; the segmentation information of the previous frame is obtained, and the segmentation information of the previous frame is stored in the short-term memory unit; the segmentation information of the first frame, the segmentation information of the periodic frame, the spatiotemporal segmentation information, and the cumulative prototype feature constitute the compressed memory unit.

[0153] In one specific embodiment, the matching module includes a global matching module, a local matching module, and a compressed matching module; the enhanced embedding feature of the current frame is matched with the segmentation information in the long-term memory unit to obtain the global segmentation embedding tensor of the current frame; the enhanced embedding feature of the current frame is matched with the segmentation information in the short-term memory unit to obtain the local segmentation embedding tensor of the current frame; the enhanced embedding feature of the current frame is clustered, propagated, and reconstructed with the segmentation information in the compressed memory unit to obtain the reconstructed segmentation tensor of the current frame; the global segmentation embedding tensor of the current frame, the local segmentation embedding tensor of the current frame, and the reconstructed segmentation tensor of the current frame are combined to obtain the segmentation embedding tensor.

[0154] In one specific embodiment, the enhanced embedding features of the current frame are clustered, propagated and reconstructed with the segmentation information in the compressed memory unit to obtain the reconstructed segmentation tensor of the current frame, specifically comprising:

[0155] The compressed memory unit comprises the segmentation information of the first frame, the segmentation information of the periodic frame, the spatiotemporal segmentation information and the accumulated prototype features; the clustering operation in the compressed matching module is performed on the segmentation information of the first frame, the segmentation information of the periodic frame, the spatiotemporal segmentation information and the accumulated prototype features to obtain the prototype features and the prototype embedding tensor; the propagation operation and the reconstruction operation in the compressed matching module are performed on the enhanced embedding features of the current frame, the prototype features and the prototype embedding tensor to obtain the reconstructed segmentation tensor of the current frame.

[0156] In one specific embodiment, step 204 specifically comprises feature decoding the segmentation embedding tensor by a feature decoding module to obtain the video target segmentation result, and a result output module stores or transmits the video target segmentation result.

[0157] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM) and memory bus dynamic RAM (RDRAM).

[0158] While the preferred embodiments in the application have been described, additional modifications and changes can occur to those skilled in the art once they learn of the basic creative principles contained herein. Therefore, the above disclosure is intended to be taken as illustrative only and not as limiting the scope of the application. The appended claims are intended to cover all modifications and changes as fall within the true scope and spirit of the application.

[0159] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A video object segmentation system, characterized by, The system comprises: a data input module receiving original video and target annotation information; a feature extraction module comprising a backbone network and a structural feature extraction module arranged in different stages of the backbone network, the backbone network extracting video frame embedding features of each frame in the original video, and the structural feature extraction module enhancing structural information in the video frame embedding features of the corresponding stage in the backbone network to obtain enhanced embedding features; a memory module storing segmentation information of a first frame and historical frames obtained according to the target annotation information; a matching module matching the enhanced embedding features of the current frame in the original video with the segmentation information in the memory module to obtain a segmentation embedding tensor; obtaining a video target segmentation result based on the segmentation embedding tensor; wherein the structural feature extraction module comprises a graph structure generation module and a graph feature extraction module: the graph structure generation module obtains the video frame embedding features of the current frame in the current stage of the backbone network; constructing an initial graph structure, and calculating a minimum spanning tree on the initial graph structure by an algorithm; the minimum spanning tree captures target structural shape information in the video frame embedding features to output an adjacency matrix and an edge weight matrix containing graph data with target structural information; the graph feature extraction module performs node feature coding, node position coding and edge weight coding on the graph data, uses a graph convolutional neural network to extract target structural features of the graph data, and adds the target structural features to the video frame embedding features to obtain the enhanced embedding features.

2. The video object segmentation system of claim 1, wherein, The system further comprises: the backbone network is arranged with at least four stages; the structural feature extraction module is arranged between the second stage and the third stage in the backbone network.

3. The video object segmentation system of claim 1, wherein, The memory module comprises a long-term memory unit, a short-term memory unit and a compressed memory unit; obtaining embedding features of the first frame, and calculating segmentation information of the first frame by combining the embedding features of the first frame and the target annotation information, and storing the segmentation information of the first frame in the long-term memory unit; obtaining segmentation information of the previous frame and storing the segmentation information of the previous frame in the short-term memory unit; obtaining segmentation information of the first frame, segmentation information of the periodic frame, spatiotemporal segmentation information and cumulative prototype features to constitute the compressed memory unit.

4. The video object segmentation system of claim 3, wherein, The matching module comprises a global matching module, a local matching module and a compressed matching module; matching the enhanced embedding features of the current frame with the segmentation information in the long-term memory unit to obtain a global segmentation embedding tensor of the current frame; matching the enhanced embedding features of the current frame with the segmentation information in the short-term memory unit to obtain a local segmentation embedding tensor of the current frame; clustering, propagating and reconstructing the enhanced embedding features of the current frame with the segmentation information in the compressed memory unit to obtain a reconstructed segmentation tensor of the current frame; combining the global segmentation embedding tensor of the current frame, the local segmentation embedding tensor of the current frame and the reconstructed segmentation tensor of the current frame to obtain the segmentation embedding tensor.

5. The video object segmentation system of claim 4, wherein, The enhanced embedding features of the current frame are clustered, propagated and reconstructed with the segmentation information in the compression memory unit to obtain a reconstructed segmentation tensor of the current frame, specifically including: The compression memory unit includes: segmentation information of a first frame, segmentation information of a periodic frame, spatiotemporal segmentation information and accumulated prototype features; The clustering operation in the compression matching module is performed on the segmentation information of the first frame, the segmentation information of the periodic frame, the spatiotemporal segmentation information and the accumulated prototype features to obtain prototype features and a prototype embedding tensor; The propagation operation and the reconstruction operation in the compression matching module are performed on the enhanced embedding features of the current frame, the prototype features and the prototype embedding tensor to obtain the reconstructed segmentation tensor of the current frame.

6. The video object segmentation system of claim 5, wherein, The system further includes: A feature decoding module that performs feature decoding on the segmentation embedding tensor to obtain a video target segmentation result; A result output module that stores or transmits the video target segmentation result.

7. A video object segmentation method based on the video object segmentation system according to any one of claims 1 to 6, characterized by, The method includes: Receiving an original video and target annotation information; Extracting video frame embedding features of each frame in the original video and enhancing the video frame embedding features at the corresponding stage in the backbone network to obtain enhanced embedding features; Obtaining enhanced embedding features of a current frame in the original video, and matching the enhanced embedding features of the current frame with segmentation information in a memory module to obtain a segmentation embedding tensor; Performing feature decoding on the segmentation embedding tensor to obtain the video target segmentation result.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the steps of the method in claim 7.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method in claim 7.

Images