Video instance segmentation method based on dynamic convolution decomposition of lightweight attention mechanism

Abstract

The application relates to a video instance segmentation method based on a dynamic convolution decomposition of a lightweight attention mechanism; the method first inputs a video frame, and a backbone network independently extracts a feature map of each frame in the video; then the feature map extracted by the backbone network enters an HQT encoding and decoding module, the position change of an instance in each frame is accurately positioned by combining an output head, and three prediction branches are used to supervise model training; in the application, a convolution layer of an encoding network adopts dynamic convolution decomposition, a more compact model is obtained, model training is easier, the required parameter quantity is greatly reduced, the training speed is improved, and the training time is shortened; a lightweight HQT encoding and decoding module is provided, which helps to reduce model parameters and improve efficiency; the loss function is improved, model training is more stable, the convergence speed and convergence precision are improved, and the problems of imbalance between foreground and background in samples and imbalance between foreground categories under a long tail condition are relieved.

Description

A video instance segmentation method with lightweight attention mechanism and dynamic convolution solution Technical Field

[0001] This invention relates to the field of video instance segmentation technology, and in particular to a video instance segmentation method based on a lightweight attention mechanism and dynamic convolution decomposition. Background Technology

[0002] Video instance segmentation, which evolved from image instance segmentation, is an emerging visual task that aims to simultaneously detect, classify, segment, and track object instances in a video. Compared to image instance segmentation, video instance segmentation is more challenging because it requires accurate object tracking throughout the entire video.

[0003] Since its initial definition in 2019, video instance segmentation has garnered significant attention due to its immense challenges and wide-ranging applications in video understanding, video editing, autonomous driving, and augmented reality. In the development of video instance segmentation, MaskTrack R-CNN and CrossVIS are algorithms based on Mask R-CNN that apply a tracking branch for cross-frame instance association to video instance segmentation. SipMask, on the other hand, uses spatial attention to describe instances, performing segmentation independently on each segment of the video through mask propagation or spatiotemporal embedding. The final instance sequence is generated by merging adjacent clip frames.

[0004] Both of the aforementioned paradigms require two independent steps to generate a complete sequence, which can easily lead to error accumulation during sequence merging, especially when occlusion and motion blur are present. Therefore, VisTR was the first to introduce the Transformer into video instance segmentation, combining it with an instance query mechanism to obtain a complete sequence in a single step from an input video, eliminating the possibility of error accumulation. IFC and SeqFormer are both improved algorithms based on Transformer, further optimizing VisTR's performance. SeqFormer is currently the best model in the offline paradigm, but the redundancy in the backbone feature network and the computational overhead of the multi-head attention mechanism increase computational costs, limiting the model's efficiency on high-resolution images. Summary of the Invention

[0005] To address the shortcomings of the aforementioned video instance segmentation techniques, this invention proposes a lightweight attention mechanism-based dynamic convolutional decomposition video instance segmentation method. A redesigned backbone feature extraction network, D-ResNet, is employed, whose convolutional layers use dynamic convolutional decomposition to independently extract feature maps from each frame of the video. Simultaneously, a lightweight HQT (HiLoQuery Transformer) encoding / decoding module is proposed, implementing a lightweight attention mechanism, HiLo Attention. By capturing global and local information through high- and low-frequency branches respectively, segmentation efficiency is significantly improved. Finally, the loss function is improved to enhance convergence accuracy and speed, resolving the imbalance problem between foreground categories in long-tail scenarios.

[0006] The objective of this invention is achieved as follows:

[0007] A video instance segmentation method based on a lightweight attention mechanism and dynamic convolution solution includes the following steps:

[0008] Step a: Input video frames, and the backbone network D-ResNet independently extracts the feature map of each frame in the video; specifically including the following steps:

[0009] Step a1: Input a single frame image into a dynamic convolutional decomposition layer with a kernel size of 7×7 and a stride of 2, and output the feature map size is reduced to 1 / 2 of the original feature map, denoted as f1;

[0010] Step a2: Input feature map f1 into the pooling layer, and the output feature map is reduced to half of feature map f1 again, which is denoted as feature map f2;

[0011] Step a3: Input feature map f2 into dynamic convolution decomposition layer layer1, and output feature map with the same size, denoted as feature map f3;

[0012] Step a4: Input feature map f3 into convolutional layer layer2, and output feature map size is halved, denoted as feature map f4;

[0013] Step a5: Input feature map f4 into convolutional layer layer3, and output feature map size is halved, denoted as feature map f5;

[0014] Step a6: Input feature map f5 into convolutional layer layer4, and output feature map size is halved, denoted as feature map f6;

[0015] Step a7: Input feature map f6 into the average pooling layer to reduce the size of the feature map, and denote it as feature map f7;

[0016] Step a8: Input feature map f7 into the fully connected layer for feature combination and classification, transforming the two-dimensional feature map into a one-dimensional vector, and outputting the result;

[0017] Step b: The feature maps extracted by the backbone network D-ResNet are fed into the HQT encoding and decoding module. The output head is used to accurately locate the position changes of instances in each frame. At the same time, three prediction branches are used to supervise model training.

[0018] Step b1: Perform a 1×1 dynamic convolution decomposition on the feature maps output by the backbone network to reduce its channel dimension to 256, generating new feature maps. m′ p ∈R C×H′×W′ , p∈[1, P];

[0019] Step b2: After adding fixed-position encoding, the output feature map is obtained via the Transformer encoder. The feature map has the same resolution as the input feature map;

[0020] Step b3: The instance query sequence is sequentially passed through the decoder layer to independently query the instance features of each frame feature map. After weighting and aggregating the temporal features, a box query sequence containing location information is obtained.

[0021] Step b4: Add three prediction branches after each decoder layer to supervise the training phase and obtain the output result. The decoder includes three prediction branches, and the total loss function is calculated as follows:

[0022]

[0023] in, It is the loss function for the category prediction branch. It is the loss function of the bounding box prediction branch. It is the loss function of the mask sequence prediction branch, λ class , λ box and λ mask These are parameters used to adjust the weights of these branches.

[0024] The aforementioned lightweight attention mechanism-based dynamic convolution solution for video instance segmentation...

[0025] In step a1, a single frame image is input to a dynamic convolution decomposition layer with a kernel size of 7×7. Since the stride is 2, the output feature map size is reduced to 1 / 2 of the original feature map, and the number of channels is 64, denoted as f1.

[0026] In step a2, feature map f1 is input into the pooling layer, the amount of data and parameters is compressed, the most significant features are kept, the number of channels remains unchanged, and the output feature map is reduced to half of f1 again, denoted as f2;

[0027] In step a3, feature map f2 is input to dynamic convolution decomposition layer layer1. The dynamic convolution decomposition layer with a kernel size of 1×1 is used to reduce and restore the number of parameters, thereby reducing the computational complexity and number of parameters of the network. The jump connection between solid and dashed lines is to adjust the number of channels. The jump connection is a solid line when the number of channels matches, and a dashed line otherwise. ×2 represents two similar structures connected together. After passing through layer1, the output feature map size remains unchanged, but the number of channels becomes 256, denoted as f3.

[0028] In step a4, feature map f3 is input to convolutional layer layer2. ×3 represents that there are 3 similar structures connected together. Since the stride of one of the convolutional layers is 2, the size of the output feature map is halved after passing through layer2, and the number of channels becomes 512, which is denoted as f4.

[0029] In step a5, feature map f4 is input to convolutional layer layer3. ×22 represents that there are 22 interconnected similar structures. Since the stride of one of the dynamic convolution decomposition layers is 2, the output feature map size is halved after passing through layer3, becoming 14×14, and the number of channels becomes 1024, which is denoted as f5.

[0030] In step a6, feature map f5 is input to convolutional layer layer4. ×2 represents that there are two similar structures connected together. Since the stride of one of the dynamic convolution decomposition layers is 2, the output feature map size is halved after passing through layer4, becoming 7×7, and the number of channels becomes 2048, which is denoted as f6.

[0031] In step a7, feature map f6 is input into the average pooling layer to reduce the size of the feature map, integrate the overall information, retain the background information, and reduce the size of the output feature map to 1×1 with 2048 channels, denoted as f7;

[0032] In step a8, the feature map f7 is input into the fully connected layer for feature combination and classification, transforming the two-dimensional feature map into a one-dimensional vector, and the result is output.

[0033] The aforementioned lightweight attention mechanism-based dynamic convolution decomposition video instance segmentation method, in step b, involves an HQT encoding / decoding module that generates feature maps from an encoding network. These maps are then passed through an encoder employing HiLo Attention. The self-attention head is divided into two groups, serving as a high-frequency attention layer and a low-frequency attention layer. The high-frequency attention layer encodes high frequencies through self-attention within each local window, while the low-frequency attention layer encodes low frequencies. Specifically, the average pooling value of the low-frequency key values ​​at each window and query position in the input feature map is globally focused on. High frequencies capture local details, while low frequencies focus on the global structure. Finally, the high and low frequency features are concatenated and passed through an instance query decoder. The decoder also internally performs HiLo Attention, combining the output head to accurately locate the positional changes of instances in each frame.

[0034] The aforementioned lightweight attention mechanism-based dynamic convolution decomposition video instance segmentation method, in step b2, after adding fixed-position encoding, performs HiLo Attention on the feature map to obtain the output feature map. The output feature map has the same resolution as the input feature map; where Add is a residual connection layer and LayerNorm is a normalization layer, used to solve the gradient vanishing problem caused by deep network structures and ensure the stability of data feature distribution; in order to execute the attention mechanism independently for each frame, the spatial and temporal dimensions of the feature map are preserved, rather than flattening the feature map into a single dimension.

[0035] The video instance segmentation method with dynamic convolution decomposition based on the lightweight attention mechanism described above, in step b3, decomposes the instance sequence into a sequence of P specific frames as anchors to retrieve and locate the features of the corresponding frames; the instance query sequence is then passed through the decoder layer to independently query the instance features mapped to each frame, wherein the current decoder layer uses the box sequence of the previous layer as input, and finally performs weighted summation and aggregates temporal features to obtain a box query sequence containing location information.

[0036] The video instance segmentation method with the aforementioned lightweight attention mechanism and dynamic convolution solution, in step b4, calculates the loss function of the category prediction branch as follows:

[0037]

[0038] Among them, c i The target category label is either an empty or non-empty set, σ(i) is the index flag of the instance embedding output by the decoder, and the category c is the category output by the category header. i The probability is denoted as

[0039] The aforementioned lightweight attention mechanism-based dynamic convolution decomposition video instance segmentation method, in step b4, uses the loss function of the bounding box prediction branch. use It is calculated by a linear combination of CIoU loss, as shown in the following formula:

[0040]

[0041]

[0042]

[0043] Where β1 and β2 represent the weight allocation parameters of the two loss functions, respectively. represent x represents the error obtained by taking the absolute value of the target value and the model output estimate. CIoU loss represents the intersection-union ratio between the predicted and ground truth bounding boxes, and b and b' represent the crossover ratio between the predicted and ground truth bounding boxes. gt ρ(·) represents the center point of the predicted box and the target box, respectively. ρ(·) represents the Euclidean distance, c represents the diagonal length of the smallest box that can simultaneously enclose the predicted box and the ground truth box, α is a positive trade-off parameter, and v is a constant representing the aspect ratio.

[0044] In the aforementioned lightweight attention mechanism-based dynamic convolutional decomposition video instance segmentation method, in step b4, the loss function of the mask sequence prediction branch is calculated using a linear combination of Dice and E-Focal loss, as shown in the following formula:

[0045]

[0046]

[0047]

[0048] in, Represents the Dice loss function. This represents E-Focal loss, where μ1 and μ2 represent the weighting parameters of the two loss functions, respectively. X is the set of predicted values, Y is the set of true values, and α... t It is a parameter that balances the importance of positive and negative samples, p t ∈[0,1], representing the prediction confidence score of the candidate object, parameter γ j The focusing factor representing class j is represented by a larger γ.j To mitigate the severe positive-negative imbalance in rare categories, a smaller γ is used for common categories with slight imbalance. j .

[0049] Beneficial effects:

[0050] First, the convolutional layers of the encoding network employ dynamic convolution decomposition. Utilizing dynamic channel fusion, different convolutional kernel coefficients are dynamically generated based on different input frames, and then the convolutional kernels are adaptively fused. This improves feature extraction and representation, thereby enhancing model accuracy. The biggest advantage of dynamic convolution decomposition lies in its use of a non-shared dynamic channel fusion mechanism to aggregate shared static basis vectors in a low-dimensional latent space, reducing the dimensionality of the crucial latent space and resulting in a more compact model. Furthermore, dynamic convolution decomposition alleviates the difficulty of jointly optimizing dynamic attention and static convolutional kernels, making model training easier and significantly reducing the number of parameters required, thus increasing training speed and shortening training time.

[0051] Secondly, a lightweight HQT encoding / decoding module is proposed, which implements HiLo Attention. The self-attention head is divided into two groups, serving as high-frequency attention layers and low-frequency attention layers, respectively. One group encodes high frequencies through self-attention within each local window, while the other group encodes low frequencies. Specifically, the average pooling value of the low-frequency key values ​​of each window and query position in the input feature map is globally focused on, with high frequencies capturing local details and low frequencies focusing on the global structure. Finally, the high- and low-frequency features are concatenated and passed through the instance query decoder. The decoder also performs HiLo Attention internally, combining the output head to accurately locate the positional changes of instances in each frame. The entire HiLo framework ensures low complexity, and the grouping of self-attention heads decomposes the learning parameters into two smaller matrices, which helps reduce model parameters and improve efficiency.

[0052] Third, improve the loss function. The linear combination of Smooth loss and CIoU loss can make the model training more stable and improve the convergence speed and convergence accuracy. The linear combination of Dice loss and E-Focal loss can alleviate the problem of foreground-background imbalance in the sample and the imbalance between foreground categories in the case of long tail. Attached Figure Description

[0053] Figure 1 is a flowchart of the video instance segmentation method based on the dynamic convolution solution of the lightweight attention mechanism of the present invention.

[0054] Figure 2 is a schematic diagram of the D-ResNet feature extraction network module structure;

[0055] Figure 3 is a detailed structural diagram of layer 1 of the convolutional layer in the D-ResNet feature extraction network module.

[0056] Figure 4 is a detailed structural diagram of the convolutional layer 2 in the D-ResNet feature extraction network module.

[0057] Figure 5 is a detailed structural diagram of layer 3 of the convolutional layer in the D-ResNet feature extraction network module.

[0058] Figure 6 is a detailed structural diagram of layer 4 of the convolutional layer in the D-ResNet feature extraction network module.

[0059] Figure 7 is a schematic diagram of the HQT encoding / decoding module structure. Detailed Implementation

[0060] The specific embodiments of the present invention will now be described in further detail with reference to the accompanying drawings.

[0061] The video instance segmentation method based on the lightweight attention mechanism and dynamic convolution decomposition in this specific implementation is shown in Figures 1-7, and includes the following steps:

[0062] This invention discloses a lightweight attention mechanism for video instance segmentation using dynamic convolution decomposition. The proposed FasFormer network model employs dynamic convolution decomposition and a lightweight attention mechanism, HiLo Attention, as shown in Figure 1. The network structure is simple: the backbone network D-ResNet extracts features from the input frames, while the HQT encoder / decoder module detects, tracks, and segments instances. First, the backbone network independently extracts feature maps for each frame of the video. The dynamic convolution decomposition utilizes a non-shared dynamic channel fusion mechanism to significantly reduce the dimensionality of the latent space, increase attention values, and promote the learning of corresponding channels. Next, the extracted feature maps enter the HQT encoder / decoder module, where a Transformer encoder performs HiLo Attention on the feature maps. Low-frequency attention branches capture global dependencies, while high-frequency attention branches capture fine-grained local dependencies, greatly improving efficiency. The resulting output feature maps then pass through an instance query decoder, which also performs HiLo Attention internally. Combined with the output header, this accurately locates the positional changes of instances in each frame, thus achieving effective instance segmentation.

[0063] Referring to Figure 2, during video instance segmentation, video frames are first input into the backbone network for feature extraction. The D-ResNet feature extraction network proposed in this invention uses dynamic convolution decomposition as the convolutional layer. Utilizing the principle of dynamic channel fusion, different convolutional kernel coefficients are dynamically generated based on different input frames. Then, the convolutional kernels are adaptively fused, thereby improving feature extraction and representation and enhancing model accuracy. The biggest advantage of dynamic convolution decomposition lies in its use of a non-shared dynamic channel fusion mechanism to aggregate shared static basis vectors in a low-dimensional latent space, reducing the dimensionality of the crucial latent space and resulting in a more compact model. Furthermore, dynamic convolution decomposition can alleviate the difficulty of jointly optimizing dynamic attention and static convolutional kernels, making model training easier and significantly reducing the number of parameters required, thus increasing training speed and shortening training time.

[0064] The ResNet network structure is shown in Figure 2. Let the given input video x be... v ∈R P×3×H×W For each P-frame with 3 color channels and a resolution of H×W, the backbone network D-ResNet independently extracts the feature map of each frame. For ease of representation, we assume that the size of a single frame image is 3 color channels and H×W is 224×224. In the structure diagram of Figure 2: DCD represents dynamic convolution decomposition; C = (3, 64) where C represents the number of channels, 3 is the number of input channels I, and 64 is the number of output channels O; K represents the kernel size, K = 7, which means the kernel size is 7×7; S represents stride; P represents padding; Pooling represents pooling layer; the convolutional layers of the D-ResNet network are divided into 4 layers, namely layers 1-4, which will be explained in detail later; Average Pool represents average pooling layer; FC represents fully connected layer.

[0065] The steps for feature extraction from a single frame image are as follows:

[0066] Referring to Figure 2, step a1: A single frame image of size 224×224 with 3 color channels is input to a dynamic convolutional decomposition layer with a kernel size of 7×7. Since the stride is 2, the output feature map size is reduced to 1 / 2 of the original feature map, and the number of channels is 64, denoted as f1.

[0067] Referring to Figure 2, step a2: the feature map f1 is input into the pooling layer, the amount of data and parameters is compressed, the most significant features are kept, the number of channels remains unchanged, and the output feature map is reduced to half of f1 again, denoted as f2;

[0068] Referring to Figure 3, step a3: Feature map f2 is input to convolutional layer layer 1. The dynamic convolutional decomposition layer with a kernel size of 1×1 serves to reduce and restore the number of parameters, thereby reducing the computational complexity and parameter count of the network. The jump connections between solid and dashed lines are used to adjust the number of channels; solid lines represent jump connections when the number of channels matches, and dashed lines otherwise. ×2 represents two similar structures connected together. After passing through layer 1, the output feature map size remains unchanged, but the number of channels becomes 256, denoted as f3.

[0069] Referring to Figure 4, step a4: feature map f3 is input to convolutional layer layer 2. ×3 represents that there are 3 connected similar structures. Since the stride of one of the DCD layers is 2, the output feature map size is halved after passing through layer 2, and the number of channels becomes 512, which is denoted as f4.

[0070] Referring to Figure 5, step a5: feature map f4 is input to convolutional layer layer 3. ×22 represents that there are 22 interconnected similar structures. Since the stride of one of the dynamic convolution decomposition layers is 2, the output feature map size is halved after passing through layer 3, becoming 14×14, and the number of channels becomes 1024, which is denoted as f5.

[0071] Referring to Figure 6, step a6: feature map f5 is input to convolutional layer layer 4. ×2 represents that there are two similar structures connected together. Since the stride of one of the dynamic convolution decomposition layers is 2, the output feature map size is halved after passing through layer 4, becoming 7×7, and the number of channels becomes 2048, which is denoted as f6.

[0072] Referring to Figure 2, step a7: feature map f6 is input into the average pooling layer to reduce the size of the feature map, integrate the overall information, retain the background information, and reduce the size of the output feature map to 1×1 with 2048 channels, denoted as f7;

[0073] Referring to Figure 2, step a8: the feature map f7 is input into the fully connected layer for feature combination and classification, transforming the two-dimensional feature map into a one-dimensional vector, and the result is output.

[0074] Referring to Figure 7, this invention proposes a lightweight HQT encoding / decoding module, employing lightweight HiLo Attention. The self-attention heads are divided into two groups, serving as high-frequency and low-frequency attention layers respectively. One group encodes high frequencies through self-attention within each local window, while the other group encodes low frequencies. This is achieved by globally focusing on the average pooling value of the low-frequency key values ​​at each window and query location in the input feature map. High frequencies capture local details, while low frequencies focus on the global structure. Finally, the high- and low-frequency features are concatenated, and the result is forwarded to subsequent layers. The entire HiLo framework ensures low complexity, and the grouping of self-attention heads decomposes the learning parameters into two smaller matrices, helping to reduce model parameters and improve efficiency.

[0075] (1) Transformer encoder part:

[0076] Perform a 1×1 dynamic convolution decomposition on the feature maps output by the backbone network to reduce its channel dimension to 256, generating new feature maps. m′ p ∈R C×H′×W′ , p∈[1,P]. As shown in Figure 3, after adding fixed-position encoding, the Transformer encoder performs HiLo Attention on the feature map to obtain the output feature map. It has the same resolution as the input feature map. Here, Add is a residual connection layer, and LayerNorm is a normalization layer, used to solve the gradient vanishing problem that occurs when the network structure is deep, ensuring the stability of the data feature distribution. To execute the attention mechanism independently for each frame, this paper preserves the spatial and temporal dimensions of the feature map, rather than flattening them into a single dimension.

[0077] (2) Transformer instance query decoder part:

[0078] This paper introduces a fixed number of learnable embeddings in the decoder section to query the same instance features from each frame, called instance query. Due to the variations in the appearance and location of instances, the model should focus on the distinct and precise spatial locations of each frame. To achieve this, the instance sequence is decomposed into a sequence of P specific frame bounding boxes, which serve as anchors to retrieve and locate features for the corresponding frames.

[0079] In the first decoder layer, an instance query sequence I is used. q ∈R C Independently query the instance features of the feature map for each frame:

[0080]

[0081] in, HiLoAtten represents the first encoder layer's instance query sequence for the p-th frame, and HiLoAtten represents the HiLoAttention module.

[0082] In the l-th (l>1) decoder layer, the box sequence from the previous layer is used. As input:

[0083]

[0084] Instance queries weight the sequence of bounding boxes output by each decoder layer and aggregate temporal features, where the weights are learned end-to-end during bounding box embedding:

[0085]

[0086] In the Nth d After the decoder layer, an instance query sequence and P bounding box query sequences for each instance are obtained. The instance query is a shared video-level instance representation, and the bounding box query contains location information for predicting the bounding boxes for each frame. This invention uses the instance sequence output by the last decoder layer... And box query sequence As output instance embedding and box embedding BE p ∈R N×d .

[0087] The proposed prediction branch consists of three parts, and the formula for calculating the total loss function is as follows:

[0088]

[0089] in, It is the loss function for the category prediction branch. It is the loss function of the bounding box prediction branch. It is the loss function of the mask sequence prediction branch, λ class , λ box and λ mask These are parameters used to adjust the weights of these branches.

[0090] The loss functions for these three branches are described below.

[0091] ① Loss function of the category prediction branch

[0092] The actual meaning of the output of this branch is the probability of predicting the category. The loss function of the category prediction branch is calculated as follows:

[0093]

[0094] Among them, c i The target category label is either an empty or non-empty set, σ(i) is the index flag of the instance embedding output by the decoder, and the category c is the category output by the category header. i The probability is denoted as

[0095] ②Loss function of bounding box prediction branch

[0096] Loss function of bounding box prediction branch use It is calculated by a linear combination of CIoU loss. Solved The gradient is not differentiable at 0, making it smoother and easier to converge; in the interval |x|>1, it also adopts... The linear function in the equation allows the gradient to decrease rapidly. When the input value is large in the early stage of training, it can remain relatively stable at a certain value, and when it tends to converge in the later stage, it can accelerate the backpropagation of gradients, thus effectively solving the problem. The existing problems.

[0097] When the target box completely encloses the predicted box, the GIoU loss degenerates into IoU, failing to distinguish their relative positional relationship. Therefore, this paper proposes an improved loss function, CIoU loss, which considers three important geometric factors: overlapping area, center point distance, and aspect ratio. Compared with GIoU loss, CIoU loss has a faster convergence speed and higher convergence accuracy.

[0098] The loss function for the bounding box prediction branch is calculated as follows:

[0099]

[0100]

[0101]

[0102] Where β1 and β2 represent the weight allocation parameters of the two loss functions, respectively. represent x represents the error obtained by taking the absolute value of the target value and the model output estimate. CIoU loss represents the intersection-union ratio between the predicted and ground truth bounding boxes, and b and b' represent the crossover ratio between the predicted and ground truth bounding boxes. gt ρ(·) represents the center point of the predicted box and the target box, respectively. ρ(·) represents the Euclidean distance, c represents the diagonal length of the smallest box that can simultaneously enclose the predicted box and the ground truth box, α is a positive trade-off parameter, and v is a constant representing the aspect ratio.

[0103] ③ Loss function of the masked sequence prediction branch

[0104] The loss function for the mask sequence prediction branch is calculated using a linear combination of Dice and E-Focal loss. Dice Loss mitigates the negative impact of foreground-background imbalance in the samples, where most areas of the image do not contain the target, and only a small portion does. Dice Loss training focuses more on mining foreground regions, ensuring a low false negative rate, but it suffers from loss saturation. Therefore, using Dice Loss alone often does not yield good results and requires combination, such as Dice Loss + Focal Loss.

[0105] In long-tailed data distributions, varying degrees of class imbalance exist, which is a major obstacle affecting object detection performance. Traditional Focal Loss uses the same adjustment factor to balance the training process for all classes, thus failing to address the long-tail problem. To solve this issue, this invention employs an Equalized Focal Loss function, which rebalances the loss weights of positive and negative samples for each class based on their respective imbalance levels. Specifically, the Equalized Focal Loss function uses a class-related adjustment factor that can be dynamically adjusted according to the training status of different classes.

[0106] The loss function for the masked sequence prediction branch is calculated as follows:

[0107]

[0108]

[0109]

[0110] in, Represents the Dice loss function. This represents E-Focal loss, where μ1 and μ2 represent the weighting parameters of the two loss functions, respectively. X is the set of predicted values, Y is the set of true values, and α... t It is a parameter that balances the importance of positive and negative samples, p t ∈[0,1], representing the prediction confidence score of the candidate object, parameter γ j The focusing factor representing class j is represented by a larger γ. j To mitigate the severe positive-negative imbalance in rare categories, a smaller γ is used for common categories with slight imbalance. j .

[0111] The above are all preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A video instance segmentation method based on a lightweight attention mechanism and dynamic convolution decomposition, characterized in that, Includes the following steps: Step a: Input video frames, and the backbone network D-ResNet independently extracts the feature map of each frame in the video; specifically, it includes the following steps: Step a1: Input a single frame image into a dynamic convolutional decomposition layer with a kernel size of 7×7 and a stride of 2, and reduce the output feature map size to 1 / 2 of the original feature map, denoted as Step a2: Feature Map The input pooling layer reduces the output feature map to a smaller feature map. Half of it is denoted as the feature map. Step a3: Feature Map The input is a dynamic convolution decomposition layer (layer1), and the output feature map remains the same size. This is denoted as the feature map. Step a4: Feature Map The input convolutional layer layer2 outputs a feature map of half the size, denoted as the feature map. Step a5: Feature Map The input convolutional layer layer3 outputs a feature map of half the size, denoted as the feature map. Step a6: Feature Map The input convolutional layer layer4 outputs a feature map that is halved in size, denoted as the feature map. Step a7: Feature Map Input to an average pooling layer to reduce the size of the feature map, denoted as the feature map. Step a8: Feature Map Step b: Inputting the fully connected layer, performing feature combination and classification, transforming the two-dimensional feature map into a one-dimensional vector, and outputting the result; Step b: The feature map extracted by the backbone network D-ResNet enters the HQT encoding and decoding module, combining it with the output head to accurately locate the positional changes of instances in each frame, while using three prediction branches to supervise model training; Step b1: Performing a 1×1 dynamic convolution decomposition on the feature map output by the backbone network to reduce its channel dimension to 256, generating a new feature map. ， Step b2: After adding fixed-position encoding, the output feature map is obtained via the Transformer encoder. The feature mapping has the same resolution as the input feature mapping; Step b3: The instance query sequence is sequentially passed through the decoder layer to independently query the instance features of each frame feature mapping. After weighting and aggregating temporal features, a box query sequence containing location information is obtained; Step b4: Three prediction branches are added after each decoder layer to supervise the training phase and obtain the output result. The decoder includes three prediction branches, and the total loss function is calculated as follows: in, It is the loss function for the category prediction branch. It is the loss function of the bounding box prediction branch. It is the loss function for the masked sequence prediction branch. 、 and These are parameters used to adjust the weights of these branches; in step b, the HQT encoding / decoding module includes generating a feature map from the encoding network, passing it through an encoder using HiLo Attention, and dividing the self-attention head into two groups, serving as a high-frequency attention layer and a low-frequency attention layer, respectively. The high-frequency attention layer encodes high frequencies through self-attention within each local window, while the low-frequency attention layer encodes low frequencies. Specifically, the average pooling value of the low-frequency key value of each window and query position in the input feature map is globally focused on, with high frequencies capturing local details and low frequencies focusing on the global structure. Finally, the high and low frequency features are concatenated and passed through the instance query decoder. Similarly, HiLo Attention is still performed internally in the decoder, and the output head is used to accurately locate the positional changes of instances in each frame.

2. The video instance segmentation method based on dynamic convolution decomposition using a lightweight attention mechanism according to claim 1, characterized in that, In step a1, a single frame image is input to a dynamic convolutional decomposition layer with a kernel size of 7×7. Due to the stride of 2, the output feature map size is reduced to half the original feature map size, and the number of channels is 64, denoted as […]. ； In step a2, the feature map The input pooling layer compresses the amount of data and parameters, preserving the most salient features while keeping the number of channels unchanged. The output feature map is then further reduced to... Half of, denoted as ； In step a3, the feature map The input is a dynamic convolutional decomposition layer (layer1), where the 1×1 kernel size reduces and restores the number of parameters, thus decreasing the computational complexity and parameter count of the network. The solid and dashed skip connections are used to adjust the number of channels; solid lines represent skip connections when the channel count matches, and dashed lines otherwise. ×2 represents two similar connections. After passing through layer1, the output feature map size remains unchanged, but the number of channels becomes 256, denoted as... ； In step a4, the feature map The input convolutional layer layer2, ×3 represents three interconnected similar structures. Since one of the convolutional layers has a stride of 2, the output feature map size is halved after passing through layer2, and the number of channels becomes 512. This is denoted as... In step a5, the feature map The input convolutional layer layer3, ×22, represents 22 interconnected similar structures. Since one of the dynamic convolutional decomposition layers has a stride of 2, the output feature map size is halved after layer3, becoming 14×14, and the number of channels becomes 1024. This is denoted as... ； In step a6, the feature map The input convolutional layer layer4, ×2 represents two interconnected similar structures. Since one of the dynamic convolutional decomposition layers has a stride of 2, the output feature map size is halved after layer4, becoming 7×7, and the number of channels becomes 2048. This is denoted as... In step a7, the feature map The input is an average pooling layer, which reduces the size of the feature map, integrates the overall information, preserves background information, and reduces the output feature map size to 1×1 with 2048 channels. ； In step a8, the feature map Input the fully connected layer, perform feature combination and classification, transform the two-dimensional feature map into a one-dimensional vector, and output the result.

3. The video instance segmentation method based on dynamic convolution decomposition using a lightweight attention mechanism according to claim 1, characterized in that, In step b2, after adding fixed-position encoding, the Transformer encoder performs HiLo Attention on the feature map to obtain the output feature map. The output feature map has the same resolution as the input feature map; where Add is a residual connection layer and LayerNorm is a normalization layer; in order to perform the attention mechanism independently for each frame, the spatial and temporal dimensions of the feature map are preserved, rather than flattening the feature map into a single dimension.

4. The video instance segmentation method based on dynamic convolution decomposition using a lightweight attention mechanism according to claim 1, characterized in that, In step b3, the instance sequence is decomposed into a sequence of P specific frames as anchors to retrieve and locate the features of the corresponding frames. The instance query sequence is then passed through the decoder layer to independently query the instance features of each frame feature map. The current decoder layer uses the box sequence of the previous layer as input. Finally, the time features are weighted and aggregated to obtain a box query sequence containing location information.

5. The video instance segmentation method based on dynamic convolution decomposition using a lightweight attention mechanism according to claim 1, characterized in that, In step b4, the loss function for the category prediction branch is calculated as follows: in, The target category label is either an empty set or a non-empty set. The index flags embedded for the instances output by the decoder, and the categories output by the category header. The probability is denoted as 。 6. The video instance segmentation method based on dynamic convolution decomposition using a lightweight attention mechanism according to claim 1, characterized in that, In step b4, the loss function of the bounding box prediction branch use and The calculation is performed using a linear combination of the given values, as shown in the following formula: in, and These represent the weight allocation parameters for the two loss functions, respectively. represent x represents the error obtained by taking the absolute value of the target value and the model output estimate. represent IoU represents the intersection-union ratio between predicted and ground truth bounding boxes. and These represent the center points of the predicted bounding box and the target bounding box, respectively. represents the Euclidean distance, and c represents the diagonal length of the smallest box that can simultaneously enclose both the predicted and ground truth boxes. It is a positive trade-off parameter. is a constant representing the aspect ratio.

7. The video instance segmentation method based on dynamic convolution decomposition using a lightweight attention mechanism according to claim 1, characterized in that, In step b4, the loss function of the mask sequence prediction branch is calculated using a linear combination of Dice and E-Focal loss, as shown in the following formula: in, Represents the Dice loss function. represent ， and These represent the weighting parameters for the two loss functions, respectively. X is the set of predicted values, and Y is the set of true values. It is a parameter that balances the importance of positive and negative samples. , representing the prediction confidence score of the candidate object, parameter The focus factor represents the j-th class.

Images