Video segmentation method, system, and apparatus based on semantic alignment and temporal consistency

Abstract

The application discloses a video segmentation method, system and device based on semantic alignment and time consistency, and belongs to the field of computer vision, and comprises the following steps: receiving a support image, a target mask annotation and a challenge video frame to be segmented; extracting support features and challenge features from the support image and the challenge video frame to be segmented; compressing target information in the support features to obtain a rough representation, and calculating a foreground probability of the rough representation to obtain a result of coarse-grained alignment; calculating attention sizes of the support features and the challenge features to obtain point-by-point similarity of the support features and the challenge features; compared with the prior art, the application realizes small sample video segmentation by jointly exploring target information aggregation under support guidance and time sequence information modeling; and the designed coarse-to-fine feature alignment module can extract a global target representation as coarse-grained guidance, and an optimal transport algorithm is introduced to realize accurate fine-grained alignment.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more specifically to a video segmentation method, system, and apparatus based on semantic alignment and temporal consistency. Background Technology

[0002] Video segmentation is a fundamental visual task, and deep learning-based methods have achieved great success in this field. However, current video segmentation methods heavily rely on large amounts of time-consuming and labor-intensive dense annotations. To reduce the need for manual annotation, few-shot video segmentation has attracted increasing attention. It is defined as predicting target masks with unseen categories in unlabeled video sequences (called challenge sets) with only a small number of labeled images (called support sets).

[0003] Despite significant progress in few-shot image segmentation, few-shot video segmentation remains a more challenging task due to the more complex temporal structure of videos. Furthermore, the core of few-shot learning is establishing information transfer from the support set to the challenge set; however, in few-shot video segmentation, support images and challenge videos typically differ significantly in scale, pose, and background variations, increasing the difficulty of effective information aggregation and reducing the accuracy of target segmentation. Existing few-shot video segmentation methods mainly fall into two categories: one transfers support information by learning the dense correlation between the support and challenge sets, while the other imposes implicit temporal constraints on challenge video frames during few-shot inference. The former focuses solely on the matching between the challenge and support sets, neglecting the utilization of the temporal information inherent in the video, while the latter does the opposite. Neither of these methods achieves satisfactory performance. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a video segmentation method and apparatus based on semantic alignment and temporal consistency. It also considers target semantic information aggregation guided by support sets and modeling of video temporal information. Both of these aspects are crucial for small-sample video segmentation tasks, thereby enabling more accurate segmentation predictions for moving targets in videos under the premise of small samples.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] Firstly, this application proposes a video segmentation method based on semantic alignment and temporal consistency, including:

[0007] Receive supporting images, target mask annotations, and challenge video frames to be segmented;

[0008] Extract support features and challenge features from the support image and the challenge video frame to be segmented;

[0009] The target information in the supporting features is compressed to obtain a coarse representation. The foreground probability of the coarse representation is calculated to obtain the coarse-grained alignment result. The attention magnitude of the supporting features and the challenge features is calculated to obtain the pointwise similarity between the supporting features and the challenge features.

[0010] The distribution of supporting features is determined based on the target mask annotation, and the distribution of question features is determined based on the coarse-grained alignment results. The optimal transfer matrix is ​​obtained by using point-to-point similarity as the cost matrix and the distributions of supporting features and question features as the edge distributions. The optimal transfer matrix is ​​applied to the attention to complete the information transfer and obtain the aggregated features of the final question set.

[0011] The temporal information of the previous frame of the challenge video is extracted, and the feature matrix of the challenge video is modulated based on the temporal information to obtain the foreground activation map and the background activation map. The foreground activation map and the background activation map are fused to obtain the temporal guide map, and the aggregated features are modulated through the temporal guide map. The modulated aggregated features are processed by the decoder to obtain the video segmentation result.

[0012] In some embodiments, obtaining the result of the coarse-grained alignment specifically includes the following steps:

[0013] The target information in the supporting features is compressed into a global representation.

[0014]

[0015] in is the global pooling function, and ⊙ is the Hadamard product (element-by-element multiplication). This indicates that the target mask is labeled M. s The size was adjusted to match the support feature F s Consistent;

[0016] Through the rough representation of the target p s This allows us to obtain the characteristics of the inquiry. Foreground probability of each feature point:

[0017]

[0018] Where i∈1,2,…,hw are the indices of the challenge feature points, and the resulting e i This is the result of coarse-grained alignment.

[0019] In some embodiments, the acquisition of point-by-point similarity specifically includes the following steps:

[0020] The query q is obtained through linear mapping. i , key k j Sum of values ​​v j , respectively represented as:

[0021]

[0022] in Different mapping matrices represent the mapping of challenge features and support features to another feature space used for matching, where i,j∈1,2,…,hw are the indices of the challenge feature points and support feature points, respectively;

[0023] Calculate the attention size for each query-key pair:

[0024]

[0025] Where d(·) represents the distance metric, typically using dot product similarity. As a proportionality coefficient;

[0026] The pointwise similarity S of the supporting features and the questioning features satisfies:

[0027] In some embodiments, the construction of the optimal transmission matrix specifically includes the following steps:

[0028] Define the distribution of challenge features and supporting features:

[0029]

[0030] The distribution μ of the questioning feature q It is based on the prospect probability e of the question feature. i The obtained foreground probability can be used as a coarse-grained alignment score, representing the importance of each point in the challenge set, while the distribution μ of the supporting features... s It is defined based on whether it is a foreground element;

[0031] Pointwise similarity between supporting features and questioning features As the cost matrix in the optimal transport problem, and with their respective characteristic distributions μ q ,μ s As two marginal distributions in the optimal transport problem, the alignment problem of support features and challenge features can be modeled as an optimal transport problem. The Sinkhorn algorithm can be used to efficiently obtain the optimal transport matrix T. * :

[0032]

[0033] stT1=μ q ,T T 1 = μ s

[0034] Where T is the transmission matrix, T TLet be its transpose matrix, where 1 represents a one-dimensional vector with appropriate dimensions.

[0035] In some embodiments, obtaining the aggregation features of the final challenge set specifically includes the following steps:

[0036] Apply the optimal transfer matrix to the original attention graph s i,j This allows for more accurate information transmission from support to inquiry:

[0037]

[0038] Where i,j∈1,2,…,hw are the indices of the challenge feature points and the support feature points, respectively. It is the optimal transfer matrix T * The value at coordinate (i,j) influences the final similarity score a between the j-th supporting feature and the i-th challenge feature. i,j ;

[0039] The values ​​of all hw supporting features v j via a i,j The weighted sum can reconstruct the feature z of the i-th challenge point. i The final aggregate feature Z of the question set satisfies: Z = [z1; z2; ...; z hw ].

[0040] In some embodiments, the construction of the foreground activation map and the background activation map specifically includes the following steps:

[0041] Based on the feature F of the previous frame of the interrogation video, the ratio of 30% foreground points and 50% background points is used. p Extract a set of foreground and background points with high confidence from the data;

[0042] A sparse sampling algorithm is used to select N points with the largest orthogonal distance from two sets as the foreground representations p. f and background characterization p b ;in N is the number of sampling points;

[0043] Foreground characterization p f Used to activate target-related regions to obtain a foreground activation map:

[0044]

[0045] in This represents the foreground activation map, where x, y, k, and n are the indices of height, width, channel, and foreground representation, respectively.

[0046] Background characterization p b This is used to suppress regions irrelevant to the target, resulting in a background activation map:

[0047]

[0048] This represents the background activation map, where x, y, k, and n are the indices of height, width, channel, and foreground representation, respectively.

[0049] In some embodiments, fusing the foreground activation map and the background activation map to obtain a temporal guide map, and modulating the aggregated features through the temporal guide map, specifically includes the following steps:

[0050] Foreground Activation Map and background activation image The fusion yields a timing guidance graph.

[0051] Modulating aggregated features using time-guided graphs:

[0052]

[0053] in For channel splicing operations, among which This refers to the modulated aggregation characteristics.

[0054] In some embodiments, a decoder comprising two convolutional layers and two upsampling layers is used to process the features. Processing is performed to predict the target mask. During the training phase, a linear combination of focal loss and Dice loss is used to constrain the model's prediction of the target mask during the feedforward process.

[0055] Secondly, this application proposes a video segmentation system based on semantic alignment and temporal consistency, comprising:

[0056] Receiver module: Receives supporting images, target mask annotations, and challenge video frames to be segmented;

[0057] Extraction module: Extracts support features and challenge features from the support image and the challenge video frames to be segmented;

[0058] Feature alignment module: Compresses target information in support features to obtain a coarse representation, calculates the foreground probability of the coarse representation to obtain the coarse-grained alignment result; calculates the attention magnitude of support features and challenge features to obtain the pointwise similarity of support features and challenge features; determines the distribution of support features based on target mask annotation, and determines the distribution of challenge features based on the coarse-grained alignment result; uses pointwise similarity as the cost matrix, and uses the distribution of support features and challenge features as the edge distribution to obtain the optimal transfer matrix; applies the optimal transfer matrix to the attention to complete information transfer, and obtains the aggregated features of the final challenge set.

[0059] Temporal Consistency Modulation Module: Extracts temporal information from the previous frame of the challenge video, modulates the feature matrix of the challenge video based on the temporal information to obtain a foreground activation map and a background activation map; fuses the foreground activation map and the background activation map to obtain a temporal guidance map, and modulates aggregated features through the temporal guidance map;

[0060] Output module: The decoder processes the modulated aggregated features to obtain the video segmentation result.

[0061] Thirdly, this application proposes a terminal device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. The memory stores the computer program capable of running on the processor. When the processor loads and executes the computer program, it employs a video segmentation method based on semantic alignment and temporal consistency as described in the first aspect.

[0062] The beneficial effects of this invention are:

[0063] Compared to existing methods, this approach achieves few-sample video segmentation by jointly exploring target information aggregation guided by support and temporal information modeling. The designed coarse-to-fine feature alignment module extracts a global target representation as coarse-grained guidance, and introduces an optimal transmission algorithm to achieve precise fine-grained alignment. The proposed temporal consistency modulation module utilizes historical foreground and background representations to activate target-related regions in the current frame. By simultaneously considering support and temporal information, this method significantly improves video segmentation performance under few-sample conditions. Attached Figure Description

[0064] The invention will now be further described with reference to the accompanying drawings.

[0065] Figure 1 This is a schematic diagram of the small sample video segmentation model of this application. Detailed Implementation

[0066] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0068] This invention proposes a few-sample video segmentation method based on coarse-to-fine feature alignment and temporal consistency modulation. The method consists of two parts: (1) coarse-to-fine feature alignment; and (2) temporal consistency modulation. The overall technique is as follows: Figure 1 As shown, the specific method is as follows:

[0069] (1) Feature alignment from coarse to fine. Receive the preprocessed support image I. s Target mask annotation M s and the challenge video frame I to be segmented q The data is first processed using a backbone network to extract supporting features F. s and questioning feature F q ,in Where h, w, and c represent the height, width, and channel dimension of the feature map, respectively.

[0070] In order to achieve F s and F q The feature alignment between features adopts a coarse-to-fine alignment approach: coarse-grained alignment refers to compressing the target information in the supporting features into a global representation.

[0071]

[0072] in is the global pooling function, and ⊙ is the Hadamard product (element-by-element multiplication). Indicates M s The size was adjusted to match F s Consistency. With a rough representation of the target p s This allows us to obtain the characteristics of the inquiry. Foreground probability of each feature point:

[0073]

[0074] Where i∈1,2,…,hw are the indices of the challenge feature points, and the resulting e i It is the result of coarse-grained alignment, which can be used to guide the subsequent fine-grained alignment process with global context information.

[0075] Fine-grained guidance mainly involves calculating supporting features. and questioning feature F q Point-to-point similarity between nodes. First, obtain the query q using a linear mapping. i , key k j Sum of values ​​v j , respectively represented as:

[0076]

[0077] in Different mapping matrices represent the query and support features mapped to another feature space used for matching, where i,j∈1,2,…,hw are the indices of the query and support feature points, respectively. The first step in matching query and support features is to compute the attention magnitude for each query-key pair:

[0078]

[0079] Where d(·) represents the distance metric, typically using dot product similarity. As a scaling factor, such a direct matching process in few-sample video segmentation tasks can introduce noise due to intra-class differences between the support set and the challenge set. Therefore, we use the optimal transport algorithm to enhance the foreground-to-foreground correspondence while eliminating background-to-foreground correspondence noise. To this end, we first define the distributions of the challenge features and support features:

[0080]

[0081] The distribution μ of the questioning feature q It is based on the prospect probability e of the question feature. i The obtained foreground probability can be used as a coarse-grained alignment score, representing the importance of each point in the challenge set, while the distribution μ of the supporting features... s It is defined based on whether it is foreground or background, directly filtering out the influence of background points. The point-by-point similarity of supporting features and challenge features is then used. As the cost matrix in the optimal transport problem, and with their respective characteristic distributions μ q ,μ s As two marginal distributions in the optimal transport problem, the alignment problem of support features and challenge features can be modeled as an optimal transport problem. The Sinkhorn algorithm can be used to efficiently obtain the optimal transport matrix T. * :

[0082]

[0083] stT1=μ q ,TT 1 = μ s

[0084] Where T is the transmission matrix, T T Let be its transpose, where 1 represents a vector of all ones with appropriate dimensions. The optimal transfer matrix represents the fine-grained alignment between the updated support and challenge features, applied to the original attention map s. i,j This allows for more accurate information transmission from support to inquiry:

[0085]

[0086] Where i,j∈1,2,…,hw are the indices of the challenge feature points and the support feature points, respectively. It is the optimal transfer matrix T * The value at coordinate (i,j) influences the final similarity score a between the j-th supporting feature and the i-th challenge feature. i,j The values ​​of all hw supporting features v j via a i,j The weighted sum can reconstruct the feature z of the i-th challenge point. i The final aggregate features of the challenge set are defined by Z = [z1; z2; ...; z hw ]express.

[0087] (2) Temporal Consistency Modulation. In few-sample video segmentation tasks, using only the information in the support set is insufficient; it is also necessary to jointly model the temporal consistency information implied in the challenge video. Given the features of the previous frame of the video... and the prospect probability predicted by the network Based on a ratio of 30% foreground attractions and 50% background attractions, from F p We extract sets of foreground and background points with high confidence from the data, and then use a sparse sampling algorithm to select the N points with the largest orthogonal distance from the two sets as the foreground and background representations, respectively. N is the number of sampling points.

[0088] Foreground and background representations obtained from historical frames can fully model the temporal information in the video. A foreground and background attention mechanism is used to extract this temporal information and modulate the feature matrix of the current video frame. Foreground representation p f Used to activate target-related regions:

[0089]

[0090] in This represents the foreground activation map, where x, y, k, and n are the indices of height, width, channel, and foreground representation, respectively. Specifically, x ∈ 1, 2, ..., h; y ∈ 1, 2, ..., w; k ∈ 1, 2, ..., c; and n ∈ 1, 2, ..., N. These indices serve as subscripts to represent coordinates within the matrix. Indicates the query feature F q The values ​​at coordinates (x, y, k). Background representation p b Used to suppress regions unrelated to the target:

[0091]

[0092] This represents the background activation map, where x, y, k, and n are the indices of height, width, channel, and foreground representation, respectively, and represent coordinates. The resulting foreground activation map... and background activation image The fusion yields a timing guidance graph. Used to modulate the aggregated feature Z obtained from the previous module:

[0093]

[0094] in For channel splicing operations, among which Finally, a lightweight decoder consisting of two convolutional layers and two upsampling layers is used to process the features. Processing is performed to predict the target mask. During the training phase, a linear combination of focal loss and Dice loss is used to constrain the model's prediction of the target mask during the feedforward process. The obtained target mask The final output of the model is the target segmentation result of the current frame in the video. Applying the model to subsequent frames of the video can yield the complete target segmentation result for the entire video.

[0095] This invention can be applied to video segmentation systems. The system requires only a small number of pre-set support images to segment similar targets in a large number of input videos, making it widely applicable in scenarios such as autonomous driving, video editing, and augmented reality. In implementation, it can be embedded into mobile devices as software to provide real-time video target segmentation results; alternatively, it can be installed on a backend server to process requested videos in batches.

[0096] Through testing on the YouTube-VIS data using DAN, TTI, and the method proposed in this patent, the method proposed in this patent achieves the highest accuracy performance in the industry on the small-sample video segmentation dataset YouTube-VIS.

[0097] Table 1: Comparison of experimental results on the YouTube-VIS dataset.

[0098]

[0099]

[0100] This application discloses a video segmentation system based on semantic alignment and temporal consistency, comprising:

[0101] Receiver module: Receives supporting images, target mask annotations, and challenge video frames to be segmented;

[0102] Extraction module: Extracts support features and challenge features from the support image and the challenge video frames to be segmented;

[0103] Feature alignment module: Compresses target information in support features to obtain a coarse representation, calculates the foreground probability of the coarse representation to obtain the coarse-grained alignment result; calculates the attention magnitude of support features and challenge features to obtain the pointwise similarity of support features and challenge features; determines the distribution of support features based on target mask annotation, and determines the distribution of challenge features based on the coarse-grained alignment result; uses pointwise similarity as the cost matrix, and uses the distribution of support features and challenge features as the edge distribution to obtain the optimal transfer matrix; applies the optimal transfer matrix to the attention to complete information transfer, and obtains the aggregated features of the final challenge set.

[0104] Temporal Consistency Modulation Module: Extracts temporal information from the previous frame of the challenge video, modulates the feature matrix of the challenge video based on the temporal information to obtain a foreground activation map and a background activation map; fuses the foreground activation map and the background activation map to obtain a temporal guidance map, and modulates aggregated features through the temporal guidance map;

[0105] Output module: The decoder processes the modulated aggregated features to obtain the video segmentation result;

[0106] This application also discloses a terminal device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, it employs any of the video segmentation methods based on semantic alignment and temporal consistency described in the above embodiments.

[0107] The terminal device can be a computer device such as a desktop computer, a laptop computer, or a cloud server. The terminal device includes, but is not limited to, a processor and a memory. For example, the terminal device may also include input / output devices, network access devices, and buses.

[0108] The processor can be a central processing unit (CPU). Of course, depending on the actual use, it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc., and this application does not limit it in this regard.

[0109] The memory can be an internal storage unit of the terminal device, such as a hard disk or RAM of the terminal device, or an external storage device of the terminal device, such as a plug-in hard disk, smart memory card (SMC), secure digital card (SD), or flash memory card (FC) equipped on the terminal device. Furthermore, the memory can be a combination of internal storage units and external storage devices of the terminal device. The memory is used to store computer programs and other programs and data required by the terminal device. The memory can also be used to temporarily store data that has been output or will be output. This application does not limit this.

[0110] In this terminal device, any of the video segmentation methods based on semantic alignment and temporal consistency in the above embodiments can be stored in the memory of the terminal device and loaded and executed on the processor of the terminal device for convenient use.

[0111] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A method for video segmentation based on semantic alignment and temporal consistency, characterized in that, include: Receive supporting images, target mask annotations, and challenge video frames to be segmented; Extract support features and challenge features from the support image and the challenge video frame to be segmented; The target information in the supporting features is compressed to obtain a coarse representation, and the foreground probability of the coarse representation is calculated to obtain the result of coarse-grained alignment. Calculate the attention magnitudes of supporting and challenging features to obtain the pointwise similarity between supporting and challenging features; The distribution of supporting features is determined based on the target mask annotation, and the distribution of challenge features is determined based on the coarse-grained alignment results. Using pointwise similarity as the cost matrix and the distributions of supporting features and question features as the marginal distributions, the optimal transfer matrix is ​​obtained. The optimal transfer matrix is ​​then applied to the attention to complete information transfer, resulting in the aggregated features of the final question set. Extract the temporal information of the previous frame of the challenge video, and modulate the feature matrix of the challenge video based on the temporal information to obtain the foreground activation map and the background activation map; fuse the foreground activation map and the background activation map to obtain the temporal guidance map, and modulate the aggregated features through the temporal guidance map; The video segmentation result is obtained by processing the modulated aggregated features through a decoder.

2. The video segmentation method based on semantic alignment and temporal consistency according to claim 1, characterized in that, The acquisition of the coarse-grained alignment result specifically includes the following steps: The target information in the supporting features is compressed into a global representation. : in It is a global pooling function. It is the Hadamard product (element-by-element multiplication). This indicates that the target mask is labeled. The dimensions were adjusted to match the support features. Consistent; Through the rough representation of the target , obtain the characteristics of the inquiry Foreground probability of each feature point: in It is the index of the query feature points, which is obtained This is the result of coarse-grained alignment.

3. The video segmentation method based on semantic alignment and temporal consistency according to claim 1, characterized in that, The acquisition of point-by-point similarity specifically includes the following steps: The query is obtained through linear mapping. ,key Sum , respectively represented as: in Different mapping matrices represent different ways of mapping the challenge features and support features to another feature space used for matching. These are the indices of the challenge feature points and the support feature points, respectively. Calculate the attention size for each query-key pair: in Distance is represented by dot product similarity. As a proportionality coefficient; Pointwise similarity between supporting features and questioning features satisfy: .

4. The video segmentation method based on semantic alignment and temporal consistency according to claim 1, characterized in that, The construction of the optimal transmission matrix specifically includes the following steps: Define the distribution of challenge features and supporting features: The distribution of interrogation features It is based on the prospect probability of the question features. The resulting foreground probability can be used as a coarse-grained alignment score, characterizing the importance of each point in the challenge set, while the distribution of supporting features... It is defined based on whether it is a foreground element; Pointwise similarity between supporting features and questioning features As the cost matrix in the optimal transport problem, and their respective characteristic distributions are... As two marginal distributions in the optimal transport problem, the alignment problem of support features and challenge features is modeled as an optimal transport problem, and the Sinkhorn algorithm is used to efficiently obtain the optimal transport matrix. : in For the transmission matrix, Let it be its transpose matrix. This represents a uniform vector with appropriate dimensions.

5. The video segmentation method based on semantic alignment and temporal consistency according to claim 1, characterized in that, The acquisition of the aggregated features of the final challenge set specifically includes the following steps: Apply the optimal transfer matrix to the original attention graph This allows for more accurate information transmission from support to inquiry: in These are the indices of the challenge feature points and the support feature points, respectively. It is the optimal transfer matrix The median coordinate is The value of affects the first The first supporting feature and the first The final similarity score of each question feature ; All The value of each supporting feature pass The weighted average can reconstruct the first... Features of each inquiry point The aggregation characteristics of the final question set .

6. The video segmentation method based on semantic alignment and temporal consistency according to claim 1, characterized in that, The construction of the foreground activation map and the background activation map specifically includes the following steps: Based on the characteristics of the previous frame of the interrogation video, the ratio of 30% foreground points and 50% background points is used. Extract a set of foreground and background points with high confidence from the data; A sparse sampling algorithm is used to select N points with the largest orthogonal distance from two sets as the foreground representations. and background representation ;in N is the number of sampling points; Foreground representation Used to activate target-related regions to obtain a foreground activation map: in This represents a foreground activation map. These are indexes for height, width, channel, and foreground representation, respectively; Background characterization This is used to suppress regions irrelevant to the target, resulting in a background activation map. This represents the background activation image. These are the indexes for height, width, channel, and foreground representations, respectively.

7. The video segmentation method based on semantic alignment and temporal consistency according to claim 1, characterized in that, The process of fusing the foreground activation map and the background activation map to obtain a temporal guide map, and then modulating the aggregated features using the temporal guide map, specifically includes the following steps: Foreground Activation Map and background activation image The fusion yields a timing guidance graph. ; Modulating aggregated features using time-guided graphs: in For channel splicing operations, among which ; This refers to the modulated aggregation characteristics.

8. The video segmentation method based on semantic alignment and temporal consistency according to claim 1, characterized in that, A decoder consisting of two convolutional layers and two upsampling layers is used to process the features. Processing is performed to predict the target mask. ; During the training phase, a linear combination of focal loss and Dice loss is used to constrain the model's prediction of the target mask during the feedforward process. .

9. A video segmentation system based on semantic alignment and temporal consistency, characterized in that, The system is used to implement the video segmentation method based on semantic alignment and temporal consistency as described in any one of claims 1 to 8, and the system comprises: Receiver module: Receives supporting images, target mask annotations, and challenge video frames to be segmented; Extraction module: Extracts support features and challenge features from the support image and the challenge video frames to be segmented; Feature alignment module: Compresses target information in support features to obtain a coarse representation, calculates the foreground probability of the coarse representation to obtain the coarse-grained alignment result; calculates the attention magnitude of support features and challenge features to obtain the pointwise similarity of support features and challenge features; determines the distribution of support features based on target mask annotation, and determines the distribution of challenge features based on the coarse-grained alignment result; uses pointwise similarity as the cost matrix, and uses the distribution of support features and challenge features as the edge distribution to obtain the optimal transfer matrix; applies the optimal transfer matrix to the attention to complete information transfer, and obtains the aggregated features of the final challenge set. Temporal Consistency Modulation Module: Extracts temporal information from the previous frame of the challenge video, modulates the feature matrix of the challenge video based on the temporal information to obtain a foreground activation map and a background activation map; fuses the foreground activation map and the background activation map to obtain a temporal guidance map, and modulates aggregated features through the temporal guidance map; Output module: The decoder processes the modulated aggregated features to obtain the video segmentation result.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, The memory stores a computer program that can run on a processor. When the processor loads and executes the computer program, it employs the video segmentation method based on semantic alignment and temporal consistency as described in any one of claims 1 to 8.

Images