A video object segmentation method and device with implicit motion compensation

Abstract

The present invention discloses a method and device for video object segmentation using implicit motion compensation, which belongs to the field of video processing technology and is used to solve the technical problems of slow motion compensation network calculation, content redundancy, and low accuracy in unsupervised video object segmentation in existing video processing. The method comprises: extracting consecutive frames from a video to be processed, and extracting features from the consecutive frames using a shared feature encoder to obtain embedded features; calculating a similarity matrix on the embedded features to obtain a similarity matrix; normalizing the similarity matrix to obtain attention-enhanced features; skipping the embedded features of each layer to connect to a higher layer for prediction guidance to generate a final feature; aligning the final feature of the central frame with the final feature of each adjacent frame to obtain an aligned feature; fusing the aligned features to obtain a fused feature; performing segmentation prediction on the fused feature to obtain a predicted segmentation mask, so as to segment the video object using the predicted segmentation mask.

Description

Technical Field

[0001] The present application relates to the field of video processing, and in particular to a method and device for implicit motion compensation video object segmentation. Background Art

[0002] In recent years, with the significant advancement of deep learning in the field of computer vision, many existing methods have achieved impressive results, often leveraging convolutional neural networks to directly learn nonlinear mapping functions of images, thanks to their powerful learning capabilities and end-to-end training approach. Multi-frame methods primarily focus on the temporal dependencies between consecutive frames. Previous multi-frame methods have mostly used sparse priors and total variation to perform sub-pixel motion registration and model temporal dependencies. However, this iterative approach involves expensive computational costs and limits its ability to handle large and complex motions.

[0003] In recent years, numerous methods have combined the representational power of deep learning with inter-frame consistency, modeling inter-frame correlations through bidirectional recursive structures to form motion compensation networks. These methods can learn temporal correlations without pre- and post-processing multiple frames. However, existing motion compensation networks are computationally slow. Most use a reference frame and the current frame as input to the motion compensation network to reconstruct optical flow. This results in redundant content, high computational complexity, and high cost, leading to low accuracy in unsupervised video object segmentation, which in turn affects the precision of video object extraction. Summary of the Invention

[0004] The embodiments of the present application provide a method and device for implicit motion compensation video object segmentation, which are used to solve the following technical problems: the motion compensation network in existing video processing has slow calculation, redundant content, and low accuracy in unsupervised video object segmentation.

[0005] The embodiments of this application adopt the following technical solutions:

[0006] On the one hand, an embodiment of the present application provides a method for implicit motion compensation video object segmentation, characterized in that the method includes: extracting continuous frames from the video to be processed, and extracting features from the continuous frames through a shared feature encoder to obtain embedded features; wherein the continuous frames include a central frame and several adjacent frames; performing similarity matrix calculation on the embedded features to obtain a similarity matrix; and normalizing the similarity matrix to obtain attention enhancement features; jump-connecting the embedded features of each layer to a higher level for prediction guidance to generate final features; aligning the final features of the central frame with the final features of each adjacent frame to obtain aligned features; fusing the aligned features to obtain fused features; performing segmentation prediction on the fused features to realize implicit motion compensated video object segmentation and obtain a predicted segmentation mask, so as to segment the video object through the predicted segmentation mask.

[0007] The embodiment of the present application obtains embedded features by extracting continuous frames in the video, and then normalizes the similarity matrix through calculation to obtain attention enhancement features. It then performs prediction guidance based on layer-by-layer jump connections to generate the final features of the central frame and adjacent frames, and aligns them with each other. It then performs segmentation prediction on the fused features to obtain a segmentation mask, and finally achieves the segmentation of the video object. The architecture of this segmentation method can retain the underlying details, and estimates the corresponding segmentation mask through four key modules: feature extraction module, similarity calculation module, attention propagation module, and motion compensation module, and uses the final segmentation mask to achieve the cutting of video objects. This method speeds up information processing, simplifies content processing, reduces costs, and greatly improves the accuracy of unsupervised video object segmentation through an implicit motion compensation network.

[0008] In a feasible implementation, continuous frames in the video to be processed are extracted, and features are extracted from the continuous frames through a shared feature encoder to obtain embedded features, specifically including: Extract the preset 2N+1 consecutive frames I from the video to be processed i , and select the middle frame as the center frame and the other frames as adjacent frames; wherein w*h is the spatial resolution of each frame image in the video to be processed, Δt is the frame interval, R is a real number, N is a constant, and t is time; the extracted continuous frames I i As the input of the shared feature encoder, the embedded feature is obtained Where l is the residual stage, l∈{2,3,4,5}, s is the scale, s∈{4,8,16,32}, is the feature mapping channel corresponding to the embedded feature.

[0009] In a feasible implementation, similarity matrix calculation is performed on the embedded features to obtain a similarity matrix; and the similarity matrix is normalized to obtain attention enhancement features, specifically including: extracting key features of each frame in the continuous frames based on a pre-built key encoder; performing key mapping on the center frame and the adjacent frames through the key encoder to obtain key mapping features Among them, ξ θ For the projection head of the backbone feature to the key space, is the embedding feature at the residual stage l=5; according to S=(P T ·[K i ]) T ·(Q T ·[K i ]), obtain the similarity matrix S between the consecutive frames; wherein the symbol · is a concatenation operation, as well as Both are trainable weight matrices, R is a real number, C k is the dimension and is set to 64, P T and Q T is the transposed matrix; the similarity matrix S rows between the consecutive frames are normalized by the softmax function to obtain the normalized similarity matrix Among them, S r ∈[0,1] (w′h′)*(w′h′) , w*h is the spatial resolution of each frame of the video to be processed, i, n, j are constants; based on the normalized similarity matrix S r , for the embedding features at the residual stage l=5 Calculate the weighted sum to get the attention enhancement feature Z i =V i 5 ·S r ;in,

[0010] In the embodiment of the present application, the key features of each frame are independently extracted by the key encoder in the similarity module, and the center frame is symmetrical with the adjacent frames. The obtained embedded features are input into the key encoder, and the center frame and the adjacent frames are key mapped. Then, based on the correlation between the center frame and the adjacent frames, the similarity matrix is calculated. The similarity matrix can effectively capture the global dependency between the input image feature space, and then the similarity matrix is normalized using the softmax function to achieve enhanced features. Then, the weighted sum is calculated based on the normalized similarity matrix to obtain the attention enhancement feature, and then the global dependency is transferred in a top-down manner in the attention propagation module.

[0011] In a feasible implementation manner, the similarity matrix S between consecutive frames specifically includes: according to S1=[Ki ] T ·W·[K i ], and obtain the initial similarity matrix S1 between consecutive frames; where, is a trainable weight matrix; the trainable weight matrix W is decomposed into two reversible training weight matrices as well as A similarity matrix S between the consecutive frames is obtained.

[0012] In a feasible implementation, the embedded features of each layer are connected to a higher level for prediction guidance to generate the final features, and the generation of the final features specifically includes: Get the final feature in, l∈{2,3,4} is the residual stage; up is a bilinear interpolation upsampling operation with an amplitude of 2, The residual stage projection head of the backbone features to the key space, It consists of a head convolution layer and a residual block.

[0013] is the side mask of the l+1th layer residual network.

[0014] The embodiment of the present application uses an attention propagation module to jump-connect the embedded features of each layer to a higher level for prediction guidance, adopts a global average pooling operation, and utilizes the key mapping of the embedded features of the bottom-up path horizontal connection. As the last step of the attention propagation module, two convolutional layers and a sigmoid function are used to predict the deep supervision of the predicted side mask.

[0015] In a possible embodiment, according to Calculate the side mask of the lth layer residual network Among them, the two convolutional layers Deep supervision is achieved with the function layer σ to predict the side masks.

[0016] In a feasible implementation, the final feature of the central frame is aligned with the final feature of each adjacent frame to obtain an aligned feature, specifically comprising: aligning the final feature f of the central frame with the final feature f of each adjacent frame to obtain an aligned feature. ref The final features f of adjacent frames nbr{+,-} Align to obtain the learnable parameters of the deep cascade layer

[0017] Among them, l∈{1,2,3,4} is the depth cascade layer, is the convolutional layer, is a lightweight offset generator, [·,·] is a cascade operation, and let According to the calculation of the learnable parameters of the depth cascade layer to be performed in each convolutional layer, we can obtain θ = {Δp n |n=1,...,||R||}; where θ is the learnable offset of the convolution kernel at the sampling position, and R is a real number; the final features of the central frame are aligned with the final features of each adjacent frame through the deformable convolutional network DCN to obtain the aligned features Among them, w is the weight of the convolution kernel, p0 and p n is the sampling position of the convolution kernel.

[0018] The embodiment of the present application aligns the final features of the central frame with the final features of each adjacent frame to obtain the learnable parameters of the deep cascade layer, which are then converted into learnable offsets of the convolution kernel at the sampling position, and the final alignment is performed through a deformable convolutional network to obtain the features. Using a deformable convolutional network, an adaptive deformable kernel is used to achieve feature-level alignment without deforming the image. It consists of multiple calibration operations organized in a cascade manner and follows convolution layers of the same kernel size to predict parameters. Using a four-level cascade structure, the coarsely aligned features can be gradually refined, greatly improving the alignment of the feature level.

[0019] In a feasible implementation, the alignment features are fused to obtain fused features, specifically including: obtaining the final feature f of the central frame by: ref and connection characteristics Fusion is performed to obtain the fusion feature f″=A s (f′)*f′+δ θ (A s (f′)); where

[0020] is the alignment feature, is the spatial attention parameter, δ θ is a lightweight encoder, C is the dimension, w*h is the spatial resolution of each frame of the video to be processed, and N is a constant;

[0021] is the intermediate fusion feature,

[0022] Note the parameters for time.

[0023] In a feasible implementation, segmentation prediction is performed on the fusion feature to implement implicit motion compensation video object segmentation, and a predicted segmentation mask is obtained to segment the video object through the predicted segmentation mask, specifically including: performing joint degree prediction through a loss function, performing segmentation prediction on the fusion feature, and obtaining a predicted segmentation mask in, w*h is the spatial resolution of each frame of the video to be processed.

[0024] On the other hand, an embodiment of the present application also provides a video object segmentation device with implicit motion compensation, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor so that the at least one processor can execute an implicit motion compensated video object segmentation method described in any of the above embodiments.

[0025] This application proposes an unsupervised video object segmentation method with implicit motion compensation. It uses an implicit motion compensation network to construct an encoding-decoding model, retains the underlying details, and refines the high-level global picture. Through the four major modules of feature extraction module, similarity calculation module, attention propagation module, and motion compensation module, it accelerates the information processing speed. The video object is then segmented by the obtained predicted segmentation mask, which simplifies the content processing, reduces the cost, and greatly improves the accuracy of unsupervised video object segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments described in the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0027] Figure 1 A flowchart of a video object segmentation method with implicit motion compensation provided in an embodiment of the present application;

[0028] Figure 2 An overall framework diagram of an implicit motion compensation network provided in an embodiment of the present application;

[0029] Figure 3 A framework diagram of an attention propagation module provided in an embodiment of the present application;

[0030] Figure 4 A structural diagram of a cascade alignment module provided in an embodiment of the present application;

[0031] Figure 5 A spatiotemporal compensation operation diagram provided in an embodiment of the present application;

[0032] Figure 6 A schematic structural diagram of a video object segmentation device with implicit motion compensation provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the technical solutions in this application, the following will clearly and completely describe the technical solutions in the embodiments of this application in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0034] The embodiment of the present application provides a video object segmentation method with implicit motion compensation, such as Figure 1 As shown, the method specifically includes steps S101-S106:

[0035] S101, a feature extraction module extracts consecutive frames from a video to be processed, and extracts features from a central frame and several adjacent frames in the consecutive frames through a shared feature encoder to obtain embedded features.

[0036] Specifically, according to Extract the preset 2N+1 consecutive frames I from the video to be processed i , and select the middle frame as the center frame and the other frames as adjacent frames. Among them, w*h is the spatial resolution of each frame in the video to be processed, Δt is the frame interval, R is a real number, N is a constant, and t is time. i As the input of the shared feature encoder, the embedded feature is obtained Where l is the residual stage, l∈{2,3,4,5}, s is the scale, s∈{4,8,16,32}, is the feature map channel corresponding to the embedded feature.

[0037] In one embodiment, Figure 2 The overall framework diagram of an implicit motion compensation network provided in the embodiment of the present application is as follows: Figure 2 As shown, according to the feature extraction module, the preset 2N+1 continuous frames I i Extract features, t is the center frame. t+NΔt is the next frame after the center frame, t-NΔt is the previous frame of the center frame, and then the extracted continuous frames I i As input, it is fed into a shared feature encoder (not shown in the figure) pre-trained on the ImageNet and DUTS datasets, and the last four convolutional blocks of the residual neural network (i.e. Figure 2 In the feature extraction module, the four blocks corresponding to 1 / 4, 1 / 8, 1 / 16, and 1 / 32 are used as the backbone of each stream to obtain the embedded features of the four residual stages.

[0038] S102: The similarity calculation module calculates the similarity matrix of the embedded features to obtain a similarity matrix, and normalizes the similarity matrix to obtain the attention enhancement feature.

[0039] Specifically, firstly, based on the pre-built key encoder, the key features of each frame in the continuous frames are extracted. Then, the key encoder is used to perform key mapping on the center frame and adjacent frames to obtain the key mapping features. Among them, ξ θ The projection head of the backbone feature to the key space is usually composed of a 3*3 convolution layer and then activates the ReLU function. is the embedding feature when the residual stage l = 5. According to S1 = [K i ] T ·W·[K i ], and get the initial similarity matrix S1 between consecutive frames. Among them, is a trainable weight matrix. And the trainable weight matrix W is decomposed into two reversible training weight matrices as well as Get the similarity matrix S between consecutive frames.

[0040] In one embodiment, as Figure 2 As shown in the similarity calculation module, the key encoder is used to perform key mapping on the central frame and the adjacent frames to obtain the key mapping feature K of each frame. t-NΔt , K t , K t+NΔt Then, similarity calculation is performed together with the embedded features of the last stage of the residual neural network to obtain the initial similarity matrix S1. In order to maintain the consistency between video frames, W is approximately decomposed into two reversible training weight matrices as well as Substitute the initial similarity matrix S1 to obtain the similarity matrix S.

[0041] Furthermore, after substituting two reversible training weight matrices, according to S=(P T ·[K i ]) T ·(Q T ·[K i ])=(P T ·[K i ]) T ·(Q T ·[K i ]), and obtain the similarity matrix S between consecutive frames. Among them, the symbol · is a concatenation operation, as well as Both are trainable weight matrices, R is a real number, C k is the dimension and is set to 64, P Tand Q T is the transposed matrix.

[0042] In one embodiment, the key mapping feature K of each frame i There is an embedded feature V corresponding to i , the central frame and adjacent frames represented in the embedding space, the key mapping features learn the correlation between them, and according to the embedded features The detailed information of object mask prediction is stored to obtain the similarity matrix S between consecutive frames.

[0043] Furthermore, the similarity matrix S rows between consecutive frames are normalized by the softmax function to obtain the normalized similarity matrix Among them, S r ∈[0,1] (w′h′)*(w′h′) , w*h is the spatial resolution of each frame of the video to be processed, i, n, j are constants. Finally, based on the normalized similarity matrix S r , for the embedding features at the residual stage l=5 Calculate the weighted sum to get the attention enhancement feature Z i =V i 5 ·S r .in,

[0044] As a feasible implementation, after normalizing the similarity matrix S between consecutive frames using the softmax function, the similarity matrix S based on adjacent images is obtained. r , and embed the features through efficient matrix multiplication Calculated as a weighted sum, we get the attention-enhanced feature Z i .

[0045] S103, the attention propagation module jumps through the embedded features of each layer to connect higher levels for prediction guidance and generate the final features.

[0046] Specifically, first, according to Calculate the side mask of the lth layer residual network Among them, the two convolutional layers Deep supervision is achieved with the function layer σ to predict the side masks.

[0047] Further, according to Get the final feature in, l∈{2,3,4} is the residual stage. up is a bilinear interpolation upsampling operation with an amplitude of 2. The residual stage projection head of the backbone features to the key space, It consists of a head convolution layer and a residual block. is the side mask of the l+1th layer residual network.

[0048] In one embodiment, Figure 3 This is a framework diagram of an attention propagation module provided in an embodiment of the present application, wherein: Figure 3 for Figure 2 The specific structure of the attention propagation module in the overall framework of the implicit motion compensation network. Figure 3 As shown, the attention enhancement feature Z after similarity matrix calculation is performed on the embedded features obtained in the last residual stage above t , and then through the attention map, according to as well as By using the attention propagation module, higher-level predictions gradually refine the skip connections of the finite element to accurately locate the main target. Specifically, Skip connections are flanked by higher-level predictions. The guidance, that is After guided calculation, it becomes Then by After guided calculation, we get Get the final features of the center frame and the final residual stage of the adjacent frames

[0049]

[0050] S104: The motion compensation module aligns the final feature of the central frame with the final feature of each adjacent frame to obtain an aligned feature.

[0051] Specifically, the final feature f of the center frame ref The final features f of adjacent frames nbr{+,-} Align to obtain the learnable parameters of the deep cascade layer Among them, l∈{1,2,3,4} is the depth cascade layer, is the convolutional layer, is a lightweight offset generator, [·,·] is a cascade operation, and let According to the calculation of the learnable parameters of the deep cascade layer to be performed in each convolutional layer, we can get θ={Δp n |n=1,...,||R||}. Among them, θ is the learnable offset of the convolution kernel at the sampling position, and R is a real number. Finally, the final features of the central frame are aligned with the final features of each adjacent frame through the deformable convolutional network DCN to obtain the aligned features Among them, w is the weight of the convolution kernel, p0 and p n is the sampling position of the convolution kernel.

[0052] In one embodiment, Figure 4This is a structural diagram of a cascade alignment module provided in an embodiment of the present application, and the cascade alignment module exists in Figure 2 In the motion compensation module, such as Figure 4 As shown, the final feature f of the center frame ref The final features f of adjacent frames nbr{+,-} Perform alignment connection, where C is the connection. Then, the offset generator uses the 3*3 convolution layer of the residual network DCN to reduce the number of channels of the spliced feature map, and follows the convolution layer of the same kernel size to predict the sampling parameters. Then, the final feature of the central frame is aligned with the final feature of each adjacent frame to obtain the aligned feature. Moreover, the four-level cascade structure used can gradually refine the coarsely aligned features and improve the feature-level alignment.

[0053] S105: The spatiotemporal compensation module fuses the aligned features to obtain fused features.

[0054] Specifically, the final feature f of the center frame ref and connection characteristics Fusion is performed to obtain the fusion feature f′′=A s (f′)*f′+δ θ (A s (f′)). Where,

[0055] To align features, is the spatial attention parameter, δ θ is a lightweight encoder, C is the dimension, w*h is the spatial resolution of each frame of the video to be processed, and N is a constant. is the intermediate fusion feature, Note the parameters for time.

[0056] In one embodiment, Figure 5 A spatiotemporal compensation operation diagram is provided in the embodiment of the present application. Figure 5 As shown, the final feature f of the center frame is converted to ref and connection characteristics To integrate, and To align the features, the intermediate fusion feature f′ is obtained, and finally the fusion feature f″ is obtained based on the intermediate fusion feature f′.

[0057] Among them, such as Figure 5 As shown, the connection characteristics First, we get the time attention parameter A by multiplying the connected elements.t , and calculate the similarity distance Sum and φ(f ref ) represents channel summation and sigmoid function. Spatial attention parameter A s Using pyramid design to increase the attention reception field, through A s =Up(θ2)[Maxpool(θ1(f′)),AvgPool(θ1(f′))]))+f′, and get the spatial attention parameter A s Then we get the fusion feature Among them, θ2 and θ1 are typical convolution layers, C represents the connection, and the dimension reduction is the original input channel, Maxpool and AvgPool represent the maximum convolution pooling layer and the average convolution pooling layer with an amplitude of 2, respectively, and the added spatial attention parameter A s The spatial enhancement feature has been enhanced.

[0058] S106 , performing segmentation prediction on the fused features through a segmentation module to implement implicit motion compensated video object segmentation, and obtaining a predicted segmentation mask, so as to segment the video object through the predicted segmentation mask.

[0059] Specifically, in the segmentation module of the motion compensation module, binary cross entropy loss is used at the pixel level, image quality loss is used at the patch level, and intersection-over-union loss is used at the region level to construct a loss function. The loss function is used to predict the joint degree, and the fusion features are segmented to obtain the predicted segmentation mask. in, w*h is the spatial resolution of each frame of the video to be processed.

[0060] Among them, the loss function is L bce is the binary cross entropy loss, L ssim is the image quality loss, L iou is the intersection-over-union loss, and M is the binary ground truth.

[0061] In one embodiment, in a given video to be processed, consecutive frames According to the implicit motion compensation network, the predicted segmentation mask is obtained Side mask with the lth layer residual network l∈{2,3,4,5}, and deep supervision is performed in the last four residual stages of the attention propagation module. When l=5, the total loss function is Therefore, accurate segmentation of video objects can be achieved through the predicted segmentation mask and implicit motion compensation network.

[0062] In addition, the embodiment of the present application also provides a video object segmentation device with implicit motion compensation, such as Figure 6 As shown, the implicit motion compensation video object segmentation device 600 specifically includes:

[0063] At least one processor 601 and a memory 602 in communication with the at least one processor 601. The memory 602 stores instructions that can be executed by the at least one processor 601, so that the at least one processor 601 can execute:

[0064] Extract consecutive frames from the video to be processed and extract features from the consecutive frames using a shared feature encoder to obtain embedded features. The consecutive frames include the central frame and several adjacent frames.

[0065] Calculate the similarity matrix of the embedded features to obtain the similarity matrix. And normalize the similarity matrix to obtain the attention enhancement feature.

[0066] The embedded features of each layer are skipped to connect to higher layers for prediction guidance to generate the final features;

[0067] Align the final features of the central frame with the final features of each adjacent frame to obtain aligned features;

[0068] Fuse the aligned features to obtain fused features;

[0069] Segmentation prediction is performed on the fused features to realize implicit motion compensation video object segmentation, and a predicted segmentation mask is obtained to segment the video object through the predicted segmentation mask.

[0070] This application proposes a video object segmentation method and device based on an implicit motion compensation network. The implicit motion compensation network is used to pre-build an encoding-decoding model, which can retain the underlying details and refine the high-level global picture. Then, through the four major modules of feature extraction module, similarity calculation module, attention propagation module, and motion compensation module, the information processing speed is accelerated. Finally, the video object is segmented by the obtained predicted segmentation mask, which realizes the simplified content processing, reduces the cost, and greatly improves the accuracy of unsupervised video object segmentation.

[0071] The various embodiments in this application are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from other embodiments. In particular, the description of the device is relatively simple because it is basically similar to the method embodiment. For relevant parts, refer to the partial description of the method embodiment.

[0072] The foregoing description describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0073] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the embodiments of the present application may have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included within the scope of the claims of the present application.

Claims

1. A video object segmentation method with implicit motion compensation, characterized in that: The method comprises: Extracting consecutive frames from the video to be processed, and performing feature extraction on the consecutive frames through a shared feature encoder to obtain embedded features; wherein the consecutive frames include a central frame and several adjacent frames; Performing similarity matrix calculation on the embedded features to obtain a similarity matrix; and normalizing the similarity matrix to obtain an attention enhancement feature; The embedded features of each layer are skipped to connect to higher layers for prediction guidance to generate the final features, including: according to Get the final feature in, is the residual stage; up is a bilinear interpolation upsampling operation with an amplitude of 2, The residual stage projection head of the backbone features to the key space, It consists of a head convolution layer and a residual block. is the side mask of the l+1th layer residual network, V i l is the embedded feature, i is a constant; Aligning the final feature of the central frame with the final feature of each adjacent frame to obtain an aligned feature; fusing the alignment features to obtain fused features; Segmentation prediction is performed on the fused features to implement implicit motion compensated video object segmentation, and a predicted segmentation mask is obtained, so as to segment the video object through the predicted segmentation mask.

2. The video object segmentation method with implicit motion compensation according to claim 1, characterized in that: Extract consecutive frames from the video to be processed, and perform feature extraction on the consecutive frames through a shared feature encoder to obtain embedded features, specifically including: according to Extract the preset 2N+1 consecutive frames I from the video to be processed i , and select the middle frame as the center frame and the other frames as adjacent frames; wherein w*h is the spatial resolution of each frame image in the video to be processed, Δt is the frame interval, R is a real number, N is a constant, and t is time; The extracted continuous frames I i As the input of the shared feature encoder, the embedded feature is obtained Where l is the residual stage, l∈{2,3,4,5}, s is the scale, s∈{4,8,16,32}, is the feature mapping channel corresponding to the embedded feature.

3. The video object segmentation method with implicit motion compensation according to claim 2, characterized in that: Performing similarity matrix calculation on the embedded features to obtain a similarity matrix; The similarity matrix is normalized to obtain attention enhancement features, specifically including: Extracting key features of each frame in the continuous frames based on a pre-built key encoder; The key encoder performs key mapping on the central frame and the adjacent frames to obtain key mapping features. Among them, ξ θ For the projection head of the backbone feature to the key space, is the embedding feature at the residual stage l=5; According to S=(P T ·[K i ]) T ·(Q T ·[K i ]), obtain the similarity matrix S between the consecutive frames; wherein the symbol · is a multiplication operation, as well as Both are trainable weight matrices, R is a real number, C k is the dimension and is set to 64, P T and Q T is the transposed matrix; The similarity matrix S between the consecutive frames is normalized by the softmax function to obtain a normalized similarity matrix Among them, S r ∈[0,1] (w′h′ )*( w′h′) , w*h is the spatial resolution of each frame of the video to be processed, i, n, j are constants; Based on the normalized similarity matrix S r , for the embedding features at the residual stage l=5 Calculate the weighted sum to get the attention enhancement feature Z i =V i 5 ·S r ;in, 4. The video object segmentation method with implicit motion compensation according to claim 3, characterized in that: The similarity matrix S between consecutive frames specifically includes: According to S1=[K i ] T ·W·[K i ], and obtain the initial similarity matrix S1 between consecutive frames; where, is the trainable weight matrix; Decompose the trainable weight matrix W into two reversible training weight matrices as well as A similarity matrix S between the consecutive frames is obtained.

5. The video object segmentation method with implicit motion compensation according to claim 1, characterized in that: The method further comprises: according to Calculate the side mask of the lth layer residual network Among them, the two convolutional layers Deep supervision is achieved with the function layer σ to predict the side masks.

6. The video object segmentation method with implicit motion compensation according to claim 1, characterized in that: Aligning the final features of the central frame with the final features of each adjacent frame to obtain aligned features, specifically including: The final feature f of the center frame ref The final features f of adjacent frames nbr{+,-} Align to obtain the learnable parameters of the deep cascade layer Among them, l∈{1,2,3,4} is the depth cascade layer, is the convolutional layer, is a lightweight offset generator, [·,·] is a cascade operation, and let According to the calculation of the learnable parameters of the depth cascade layer to be performed for each convolutional layer, we can get θ={Δp n |n=1,...,||R||}; where θ is the learnable offset of the convolution kernel at the sampling position, and R is a real number; The final features of the central frame are aligned with the final features of each adjacent frame through the deformable convolutional network DCN to obtain the aligned features Among them, w is the weight of the convolution kernel, p0 and p n is the sampling position of the convolution kernel.

7. The video object segmentation method with implicit motion compensation according to claim 1, characterized in that: The alignment features are fused to obtain fused features, specifically including: The final feature f of the central frame ref and connection characteristics Fusion is performed to obtain the fusion feature f′′=A s (f′)*f′+δ θ (A s (f′)); where is the alignment feature, is the spatial attention parameter, δ θ is a lightweight encoder, C is the dimension, w*h is the spatial resolution of each frame of the video to be processed, and N is a constant; in, is the intermediate fusion feature, Note the parameters for time.

8. The video object segmentation method with implicit motion compensation according to claim 1, characterized in that: Performing segmentation prediction on the fused features to implement implicit motion compensated video object segmentation, obtaining a predicted segmentation mask, and segmenting the video object using the predicted segmentation mask, specifically comprising: The joint degree is predicted by the loss function, and the segmentation prediction is performed on the fusion feature to obtain the predicted segmentation mask in, w*h is the spatial resolution of each frame of the video to be processed.

9. A video object segmentation device with implicit motion compensation, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, so as to enable the at least one processor to perform the implicit motion compensated video object segmentation method according to any one of claims 1-8.

Images