A dual-branch video matting method

Abstract

The application discloses a double-branch video matting method, and belongs to the technical field of video and image processing. The main structure of the Swin Transformer is used as an encoder module, and a shift window self-attention mechanism is used to realize efficient capture of local and global features. The coarse branch captures the overall contour and semantic information of the foreground object, strengthens the global context modeling capability, the fine branch retains high-frequency details and structural information, and enhances the semantic reasoning of the dynamic boundary. The double branch realizes deep fusion of global semantics and local details through a mixed attention up-sampling module. Finally, a hierarchical cascade structure of a progressive refinement module is used to guide multi-stage iterative optimization of the prediction result. The method effectively solves the prediction error problem caused by dynamic foreground changes in traditional methods, realizes accurate prediction of the mask while maintaining spatiotemporal consistency, and provides a more robust solution for the video matting task in a complex scene.

Description

Technical Field

[0001] This invention relates to a video matting method, and more particularly to a dual-branch video matting method, belonging to the field of image and video processing technology. Background Technology

[0002] In the current era of booming digital vision technology, video keying, as a core technology in image editing, film and television production, and human-computer interaction, is undergoing a revolutionary transformation from traditional methods to intelligent algorithms. Since the film and television industry introduced green screen keying technology in the 1960s, the method of separating foreground objects from a solid-color background has become an industry standard, creating the breathtaking special effects scenes in films such as *Star Wars* and *Avatar*. Currently, video keying is widely used in various business scenarios across industries, such as promotional advertisements on e-commerce platforms, video editing for everyday entertainment, and background replacement in online video conferencing. However, traditional green screen technology requires special venues and is costly, clearly failing to meet the increasingly diverse needs of users.

[0003] Based on the different input data, video matting tasks can be divided into two categories: video matting methods without auxiliary input and video matting methods with auxiliary input. Video matting methods without auxiliary input achieve matting through implicit modeling, with foreground objects typically set to specific categories, such as portraits or animals. Due to the lack of sufficient prior knowledge, these models can only handle specific foreground object types, making it difficult to meet diverse user matting needs. For video matting methods with auxiliary input, the auxiliary input is generally set as a triangulation or a binary mask. A triangulation divides the image to be processed into three parts: foreground region, background region, and unknown region, while a binary mask divides the image into two parts: known region and unknown region. Some early methods used frame-by-frame triangulation to achieve video matting, but this method has high manual annotation costs. Most existing methods propagate triangulation based on a small number of triangulations according to certain matching rules, or use existing segmentation models to generate binary masks in batches.

[0004] In the field of video matting, improving the accuracy of video matting and maintaining high temporal consistency are two key challenges. The edges of target objects in videos often exhibit complex shapes, such as human hair, animal fur, and semi-transparent edges. Extracting the boundaries of these objects can easily result in jagged edges, incompleteness, or background residue. Furthermore, the movement of target objects, changes in lighting, and increased background complexity in video scenes all significantly impact foreground extraction. When foreground objects move at high speeds, high-frequency details such as hair and feathers will experience motion blur, creating visual artifacts such as missing foreground elements and background residue. Summary of the Invention

[0005] To address the problems mentioned above, the technical objective of this invention is to provide a dual-branch video matting method. This invention designs an enhanced video matting model that improves feature extraction capabilities and temporal prediction stability in dynamic scenes, thereby improving the effectiveness of video matting.

[0006] This invention applies the Swing Transformer structure to video matting tasks, utilizing a shifted window self-attention mechanism to achieve cross-window interaction and expand the receptive field. A dual-branch matting network is constructed to balance global semantics and local details, improving matting accuracy. An attention-guided feature fusion module achieves deep feature aggregation, preserving original semantic information while fusing richer detailed features. The dual-branch matting network consists of a coarse-grained branch and a fine-grained branch. The dual-branch encoder adopts a hierarchical structure of the Swing Transformer, using a feature fusion module to achieve downsampling and generate a multi-scale feature pyramid. The coarse-grained branch calculates window attention between larger image patches, improving the model's ability to capture global context, while the fine-grained branch calculates window attention between smaller image patches, increasing attention to unknown regions and enhancing the model's spatiotemporal semantic reasoning ability for dynamic foreground boundaries. The two branches achieve deep feature fusion through a hybrid attention-guided upsampling module, effectively integrating global semantic information and local detailed features. The shifted window self-attention mechanism is implemented through the standard local window multi-head attention module and the shifted window multi-head attention module. The standard local window multi-head attention module divides the feature map into multiple non-overlapping windows and performs self-attention calculation independently in each window. The shifted window multi-head attention module, which is used in pairs, performs shifting operations on the windows between adjacent layers, so that information between different windows can be passed to each other, thereby enhancing the modeling ability of the model.

[0007] To achieve the above objectives, the technical solution adopted by this invention is a method for dual-branch video image matting, the implementation process of which is as follows:

[0008] Step 1: Obtain the original video sequence and the triangulation sequence. The first frame of the triangulation sequence is annotated by the user. Extract consecutive image frames from the original video sequence. ,in The number of consecutive image frames. This refers to the frame number of the image frame in the original video stream. ;

[0009] Step 2: Construct a dual-branch video matting model;

[0010] A dual-branch video matting model is constructed based on the U-Net network model, including a triangulation module, a matting module, and a progressive refinement module. The other triangulations in the triangulation sequence, except for the first frame triangulation, are predicted by the triangulation module. The matting module includes two branches, each of which includes an encoder module, a feature fusion module, and a decoder module.

[0011] The dual-branch video matting model uses the first image frame and the user-annotated first-frame tri-image from the original video sequence as initial inputs. The stitched first image frame and the user-annotated first-frame tri-image are directly input to the matting module without passing through the tri-image propagation module. The matting module, based on prior information about unknown regions provided by the tri-image, extracts image features through the encoder module and performs information interaction and semantic enhancement through the feature fusion module. The image feature pyramid obtained by the feature fusion module is input to the decoder module for coarse image matting. Mask prediction generates three different resolution prediction results, which are then iteratively optimized by a progressive refinement module to finally obtain the refined result. Mask prediction; first frame image, first frame trisection, refinement The mask prediction is then input into the tri-image propagation module as memory information. The next image frame is input into the tri-image propagation module as query information. The tri-image propagation module integrates spatiotemporal information through the spatiotemporal memory module, calculates feature similarity, and obtains the next frame's predicted tri-image through the decoder module. The next image frame and the predicted tri-image are then refined by the matting module. Mask prediction, in this way The mask prediction process loops through consecutive image frames until all frames are processed.

[0012] Step 3: Use a dual-branch video matting model to perform video sequence matting on consecutive image frames;

[0013] The continuous image frames and the user-annotated first-frame triad are input into the dual-branch video matting model. The dual-branch video matting model generates a refined first-frame image based on the first-frame image frame and the first-frame triad using the matting module. Mask prediction, then the first image frame, the first frame trisection, and the first frame refinement. The mask prediction and the next frame image are processed by the triangulation module to generate the next frame predicted triangulation. The next frame image and the predicted triangulation are then input into the matting module to generate a coarse matting image. The mask prediction is finally refined through the progressive refinement module. Mask prediction. The above process is used to iteratively process consecutive image frames until all frames are processed, thus achieving image matting of the video sequence.

[0014] Furthermore, the tri-graph propagation module constructed in step 2 includes an encoder module, a spatiotemporal memory module, and a decoder module;

[0015] The encoder module includes two types of sub-blocks, one of which is... There are two types of sub-blocks: one is a memory encoder sub-block, and the other is a query encoder sub-block. This indicates the number of frames in the memorized image; both types of sub-blocks are implemented based on the ResNet-50 network, and the input to the memorized encoder sub-block is... Frame memory image frames, triangulation, and thinning The memory information composed of mask predictions is concatenated with encoded features to obtain the memory features. The input to the query encoder module is the current image frame, which is encoded to obtain query features. The encoder module provides information at different scales to the decoder module through skip connections.

[0016] The spatiotemporal memory module maps memory features and query features into memory key-value pairs and query key-value pairs respectively using 3×3 convolutions. It calculates similarity using non-local matching with the memory and query keys, retrieves memory values ​​based on the calculated similarity, and concatenates the retrieved memory and query values ​​along the channel dimension before inputting them into the decoder module for decoding. The decoder module uses multiple residual blocks and upsampling blocks to output a predicted trilateration.

[0017] Furthermore, the coarse-grained branches of the matting network module constructed in step 2 include a coarse-branch encoder module, a feature fusion module, and a coarse-branch decoder module;

[0018] The coarse-branch encoder module uses the Swing Transformer as the backbone network to construct hierarchical feature representations and generate feature maps at different scales. This module uses image frames... And predicting the three-part chart As input, where This indicates the current image frame number being processed. The input image is a 4×4 block segmented by the embedding layer. Each self-attention block serves as a sub-block of the coarse-branch encoder module. The self-attention block calculates local attention and performs cross-window interactions through a shift-window attention mechanism to obtain the image feature pyramid. ,in This represents the image feature map output by the coarse-branch encoder module. Indicates the current floor number. This indicates the index of the feature map in the current layer;

[0019] The attention-guided feature fusion module receives the image feature pyramid output by the encoder. The current-level features are fused with the previous-level features through a hybrid attention upsampling module, which retains the original semantic information while fusing richer detailed features. The processing flow of the feature fusion module is shown in formulas (1) and (2), where, Indicates the first The first level Each feature map This represents the image feature map output by the coarse-branch encoder module. Indicates the current floor number. This indicates the index of the feature map in the current layer. This indicates a hybrid attention upsampling operation. Indicates the first The first level The residual features are output after the feature maps are processed by a hybrid attention upsampling operation. The fused features are then processed through skip connections to reduce the number of channels, compressing the high-dimensional features to a suitable dimension and avoiding feature redundancy. The dimensionality-reduced features are then input into the decoder to restore the resolution layer by layer.

[0020]

[0021]

[0022] The coarse-branch decoder consists of four residual upsampling blocks and three prediction heads, enabling efficient mapping from deep features to high-resolution transparency maps. Each sub-block introduces skip connections to preserve detail, while the prediction heads predict feature maps at different scales, producing three different resolution side outputs for subsequent progressive refinement modules.

[0023] Furthermore, the fine-grained branching module of the matting network constructed in step 2 includes a fine-branch encoder module, a feature fusion module, and a fine-branch decoder module;

[0024] The fine-branch encoder module uses the Swing Transformer as the backbone network to construct hierarchical feature representations and generate feature maps at different scales. This module uses image frames... And predicting the three-part chart As input, where This indicates the current image frame number being processed. The input image is segmented into 2×2 blocks after being processed by the embedding layer, allowing for more precise capture of subtle changes in the image and further enhancing the model's ability to perceive image details and complex textures. Each self-attention block serves as a sub-block of the fine-branch encoder module. The self-attention block calculates local attention and performs cross-window interactions through a shift-window attention mechanism to obtain the image feature pyramid. ,in This represents the image feature map output by the fine-branch encoder module. Indicates the current floor number. This indicates the index of the feature map in the current layer;

[0025] The attention-guided feature fusion module receives the image feature pyramid output by the encoder. The current-level features and the previous-level features are fused through a hybrid attention upsampling module, which retains the original semantic information while fusing richer detailed features. The feature fusion process is shown in formulas (3) and (4), where, Indicates the first The first level Each feature map This represents the image feature map output by the fine-branch encoder module. Indicates the current floor number. This indicates the index of the feature map in the current layer. This indicates a hybrid attention upsampling operation. Indicates the first The first level The residual features are output after the feature maps are processed by a hybrid attention upsampling operation. The fused features are then processed through skip connections to reduce the number of channels, compressing the high-dimensional features to a suitable dimension and avoiding feature redundancy. The dimensionality-reduced features are then input into the decoder to restore the resolution layer by layer.

[0026]

[0027]

[0028] The fine-branch decoder consists of four residual upsampling blocks and three prediction heads, achieving efficient mapping from deep features to high-resolution transparency maps. Each sub-block introduces skip connections to preserve detail, while the prediction heads predict feature maps at different scales, producing three different resolution side outputs for subsequent progressive refinement modules.

[0029] Furthermore, the progressive refinement module built in step 2 enables progressive optimization of the cutout results from coarse to fine.

[0030] The coarse-branch and fine-branch decoders generate feature layers with output strides of 8, 4, and 1, respectively. The feature layer with a stride of 8 preserves the global semantic information of the image, corresponding to coarse-grained spatial awareness. This feature layer, after passing through the current prediction head, generates a coarse prediction. A feature layer with a stride of 4 achieves a balance between semantics and detail, capturing medium-scale structural information. This feature layer, after passing through the prediction head of this layer, generates a coarse prediction. The feature layer with a stride of 1 reverts to pixel-level resolution, containing the finest texture and edge details. This feature layer, after being processed by the prediction head of this layer, generates a coarse prediction. A rough prediction , , As side outputs, these constitute three levels of prediction results, from coarse to fine, corresponding to the optimization stages of semantic differentiation, structural restoration, and detail enhancement, respectively. These three levels of prediction results are input into the progressive refinement module, and an adaptive fusion of the prediction results is achieved through a hierarchical self-guided mechanism. This allows different levels to focus on refining their respective most effective regions, guiding the network to generate refined data. Predictive mask.

[0031] Furthermore, the weighted loss constructed in step 2 enhances the model's decision-making ability at fuzzy boundaries;

[0032] The coarse-grained branch has a larger receptive field, enabling it to capture the overall contour and semantic information of foreground objects, determine the approximate range and overall structure of foreground objects in the image, and provide preliminary global semantic guidance for image matting. The fine-grained branch retains high-frequency details and structural information, enhancing the model's ability to handle complex edges. Based on the characteristics of the coarse-grained and fine-grained branches, the features generated by the coarse-grained branch are refined... Mask prediction calculates training loss Weighted by weight w1; refining the fine-grained branch generation. Mask prediction calculates training loss By weight Weighted average to obtain the total loss As shown in formula (5), the network is guided to learn global semantic information while paying attention to detailed information such as object edges.

[0033]

[0034] Compared to existing technologies, the significant difference of this invention lies in the following: Existing technologies mostly employ a single-branch structure, making it difficult to balance global and local aspects; the dual-branch matting network proposed in this invention focuses on capturing global semantic information through a coarse-grained branch. This branch can grasp high-level information such as the category, location, and relationships of objects in the image as a whole, providing global semantic guidance for matting; the fine-grained branch focuses on extracting local detail features, clearly presenting detailed information such as object edges and textures, thus improving matting accuracy. The two branches cooperate and complement each other, effectively balancing global semantics and local details. In dynamic scenes, it can simultaneously ensure semantic consistency and detail accuracy, thereby improving matting accuracy.

[0035] Compared with the prior art, the present invention has the following beneficial effects.

[0036] This invention proposes a dual-branch video matting model, employing the Swing Transformer as the backbone network and utilizing a shift-window self-attention mechanism to fuse local and global features at different scales. A dual-branch encoder is constructed to achieve hierarchical feature modeling, forming complementary multi-granularity feature representations. The feature fusion module enhances the spatial positioning accuracy of object boundaries through spatial attention and optimizes the weight allocation of cross-granularity features using channel attention, achieving efficient interaction of spatiotemporal information and semantic enhancement, thus improving the feature representation capability of complex boundaries and weakly textured regions. The progressive refinement module performs multi-stage iterative optimization of the prediction results through a hierarchical cascade structure. In challenging scenarios such as motion blur and occlusion changes, cross-stage feature fusion and error correction effectively ensure the temporal consistency of video sequence matting. Compared with existing technologies, this invention only requires a single user-annotated tri-image to complete the matting of the entire video sequence. It exhibits strong feature extraction capabilities and high-precision matting effects even in scenarios with insufficient prior knowledge, providing a high-efficiency video matting solution for low-annotation-cost scenarios and offering technical support for rapid video editing, virtual scene synthesis, and other fields. Attached Figure Description

[0037] Figure 1 This is a structural diagram of the dual-branch video matting model provided by the present invention.

[0038] Figure 2 This is a structural diagram of the coarse-grained branch module of the matting network provided by the present invention.

[0039] Figure 3 The structure diagram of the attention-guided feature fusion module provided by the present invention.

[0040] Figure 4 The residual upsampling block structure diagram in the coarse branch and fine branch decoder provided by the present invention.

[0041] Figure 5 This is a structural diagram of the fine-grained branch module of the matting network provided by the present invention.

[0042] Figure 6 The diagram shows the structure of the hybrid attention upsampling module provided by this invention.

[0043] Figure 7 This is a diagram of the progressively refined module structure provided by the present invention. Detailed Implementation

[0044] The implementation method of the present invention will be further described in detail below with reference to the accompanying drawings. The process of a dual-branch video matting model proposed in this invention is as follows: Figure 1 As shown, the specific steps include:

[0045] Step 1: Obtain the original video sequence and the triangulation sequence. The first frame of the triangulation sequence is annotated by the user. Extract consecutive image frames from the original video sequence. ,in The number of consecutive image frames. This refers to the frame number of the image frame in the original video stream. ;

[0046] Step 2: Construct a dual-branch video matting model;

[0047] A two-branch video matting model is constructed based on the U-Net network model, including a triangulation module, a matting module, and a progressive refinement module. The other triangulation frames in the triangulation sequence, excluding the first frame's triangulation frame, are predicted by the triangulation module. The structure of the two-branch video matting model is as follows: Figure 1 As shown; the matting module includes two branches, each of which includes an encoder module, a feature fusion module, and a decoder module;

[0048] The dual-branch video matting model proposed in this invention uses the first image frame and the user-annotated first-frame tri-image from the original video sequence as initial inputs. The stitched first image frame and the user-annotated first-frame tri-image are directly input to the matting module without passing through the tri-image propagation module. The matting module extracts image features through the encoder module based on the prior information about unknown regions provided by the tri-image, and the feature fusion module realizes information interaction and semantic enhancement. The image feature pyramid obtained by the feature fusion module is input to the decoder module to achieve coarse-grained processing. Mask prediction generates three different resolution prediction results, which are then iteratively optimized by a progressive refinement module to finally obtain the refined result. Mask prediction. First image frame, first frame trisection, and refinement. The mask prediction is then input into the tri-image propagation module as memory information. The next image frame is input into the tri-image propagation module as query information. The tri-image propagation module integrates spatiotemporal information through the spatiotemporal memory module, calculates feature similarity, and obtains the next frame's predicted tri-image through the decoder module. The next image frame and the predicted tri-image are then refined by the matting module. Mask prediction, in this way The mask prediction process loops through consecutive image frames until all frames are processed.

[0049] Step 2.1: Construct the tri-graph propagation module, including the encoder module, the spatiotemporal memory module, and the decoder module;

[0050] The encoder module includes two types of sub-blocks, one of which is... There are two types of sub-blocks: one is a memory encoder sub-block, and the other is a query encoder sub-block. This indicates the number of frames in the memorized image; both types of sub-blocks are implemented based on the ResNet-50 network, and the input to the memorized encoder sub-block is... Frame memory image frames, triangulation, and thinning The memory information composed of mask predictions is concatenated with encoded features to obtain the memory features. The input to the query encoder module is the current image frame, which is encoded to obtain query features. The encoder provides information at different scales to the decoder through skip connections.

[0051] The spatiotemporal memory module maps memory features to memory keys using 3×3 convolutions. and memory value ,in Indicates key. Represents the value. This represents memory. The memory key determines the semantic granularity of feature matching, while the memory value retains detailed information about the original features. The spatiotemporal memory module maps query features to query keys through another 3×3 convolution. and query value ,in Indicates key. Represents the value. This represents a query. Then, non-local matching is used to calculate the query key. and memory key The similarity score is shown in formula (1), where, Indicates the position as Memory key, Indicates the position as The search key, Indicates the position as The memory keys and their positions are The dot product similarity of the query keys. The obtained similarity scores are normalized to obtain the position. Memory features for location Attention weight matrix of query features As shown in Equation (2), the attention weight determines the importance of the memory to the query. Based on this, the memory value is retrieved, and the retrieved memory value and the query value are concatenated along the channel dimension and input into the decoder for decoding. The decoder uses multiple residual blocks and upsampling blocks to output the predicted trilateration.

[0052]

[0053]

[0054] Step 2.2: Construct the coarse-grained branches of the image matting network module, including a coarse-branch encoder module, a feature fusion module, and a coarse-branch decoder module. The module structure is as follows: Figure 2 As shown;

[0055] The coarse-branch encoder module uses the Swing Transformer as the backbone network to construct hierarchical feature representations and generate feature maps at different scales. This module uses image frames... And predicting the three-part chart As input, where This indicates the current image frame number being processed. The input image is a 4×4 block segmented by the embedding layer. Each self-attention block serves as a sub-block of the coarse-branch encoder module. The self-attention block calculates local attention and performs cross-window interactions through a shift-window attention mechanism to obtain the image feature pyramid. ,in This represents the image feature map output by the coarse-branch encoder module. Indicates the current floor number. This indicates the index of the feature map in the current layer;

[0056] In this embodiment, a four-layer feature pyramid is constructed, using 2, 2, 6, and 2 self-attention blocks sequentially from top to bottom. For a size of... The input image, each layer outputs at a scale of , , , The feature map, where Indicates the height of the input image. This represents the width of the input image. The self-attention block is built based on the Swing Transformer block structure and is used in pairs. It contains a standard local window multi-head attention module and a shifted window multi-head attention module. The former divides the feature map into multiple non-overlapping windows and performs self-attention calculation independently within each window. The latter performs a shift operation on the windows between adjacent layers, enabling information to be passed between different windows, thereby reducing computational costs while enhancing the model's long-distance modeling capability.

[0057] The attention-guided feature fusion module receives the feature pyramid output by the encoder. The current level features are fused with the previous level features through a hybrid attention upsampling module, which preserves the original semantic information while incorporating richer detailed features, such as... Figure 3 As shown. The feature fusion processing flow is shown in formulas (3) and (4), where, Indicates the first The first level Each feature map This represents the image feature map output by the coarse-branch encoder module. Indicates the current floor number. This indicates the index of the feature map in the current layer. This indicates a hybrid attention upsampling operation. Indicates the first The first level The output after processing a feature map through a hybrid attention upsampling operation. The structure of the hybrid attention upsampling module is as follows: Figure 6 As shown, this module receives two feature maps with different resolutions. and The shapes are respectively and , This represents the number of channels in the feature map. Indicates the height of the input image. This represents the width of the input image. Feature map. and The refined feature map is obtained after two 3×3 convolutional layers. and The shape remains unchanged. This set of feature maps is input into two branches: spatial attention and channel attention. Through the organic combination of these two branches, adaptive weighting and efficient fusion of features at different levels are achieved. Finally, a 1×1 convolutional layer is used to reduce the number of channels and capture cross-channel information, resulting in the... The first level Residual features of each feature map , shape is The residual features of the current level are added to the original features to obtain the next feature map. , shape is The fused features are then processed through skip connections to reduce the number of channels, compressing the high-dimensional features to a suitable dimension and avoiding feature redundancy. The dimensionality-reduced features are then input into the decoder to restore the resolution layer by layer.

[0058]

[0059]

[0060] The coarse-branch decoder consists of four residual upsampling blocks and three prediction heads, achieving efficient mapping from deep features to high-resolution transparency maps. Each sub-block introduces skip connections to preserve detail information, and the sub-block structure is as follows: Figure 4 As shown, the prediction heads after layers 2, 3, and 4 of the decoder predict feature maps at different scales, producing resolutions of [resolution value missing]. , The side output of 1 is used for subsequent progressive refinement modules.

[0061] Step 2.3: Construct the fine-grained branching module of the image matting network, including the fine-branch encoder module, the feature fusion module, and the fine-branch decoder module. The module structure is as follows: Figure 5 As shown;

[0062] The fine-branch encoder module uses the Swing Transformer as the backbone network to construct hierarchical feature representations and generate feature maps at different scales. This module uses image frames... And predicting the three-part chart As input, where This indicates the current image frame number being processed. The input image is segmented into 2×2 blocks after being processed by the embedding layer, allowing for more precise capture of subtle changes in the image and further enhancing the model's ability to perceive image details and complex textures. Each self-attention block serves as a sub-block of the fine-branch encoder module. The self-attention block calculates local attention and performs cross-window interactions through a shift-window attention mechanism to obtain the image feature pyramid. ,in This represents the image feature map output by the fine-branch encoder module. Indicates the current floor number. This indicates the index of the feature map in the current layer;

[0063] In this embodiment, a four-layer feature pyramid is constructed, using 2, 2, 6, and 2 self-attention blocks sequentially from top to bottom. For a size of... The input image, each layer outputs at a scale of , , , The feature map, where Indicates the height of the input image. This represents the width of the input image. The self-attention block is built based on the Swing Transformer block structure and is used in pairs. It contains a standard local window multi-head attention module and a shifted window multi-head attention module. The former divides the feature map into multiple non-overlapping windows and performs self-attention calculation independently within each window. The latter performs a shift operation on the windows between adjacent layers, enabling information to be passed between different windows, thereby reducing computational costs while enhancing the model's long-distance modeling capability.

[0064] The attention-guided feature fusion module receives the feature pyramid output by the encoder. The current level features are fused with the previous level features through a hybrid attention upsampling module, which preserves the original semantic information while incorporating richer detailed features, such as... Figure 3 As shown. The feature fusion processing flow is shown in formulas (5) and (6), where, Indicates the first The first level Each feature map This represents the image feature map output by the fine-branch encoder module. Indicates the current floor number. This indicates the index of the feature map in the current layer. This indicates a hybrid attention upsampling operation. Indicates the first The first level The residual features are output after the feature maps are processed by a hybrid attention upsampling operation. The fused features are then processed through skip connections to reduce the number of channels, compressing the high-dimensional features to a suitable dimension and avoiding feature redundancy. The dimensionality-reduced features are then input into the decoder to restore the resolution layer by layer.

[0065]

[0066]

[0067] The fine-branch decoder consists of four residual upsampling blocks and three prediction heads, achieving efficient mapping from deep features to high-resolution transparency maps. Each sub-block introduces skip connections to preserve detail information, and the sub-block structure is as follows: Figure 4 As shown, the prediction heads after layers 1, 2, and 4 of the decoder predict feature maps at different scales, producing resolutions of [resolution value missing]. , The side output of 1 is used for subsequent progressive refinement modules.

[0068] Step 2.4: Construct a progressive refinement module to achieve progressive optimization of the cutout results from coarse to fine;

[0069] The coarse-branch and fine-branch decoders generate feature layers with output strides of 8, 4, and 1, respectively. The feature layer with a stride of 8 preserves the global semantic information of the image, corresponding to coarse-grained spatial awareness. This feature layer, after passing through the current prediction head, generates a coarse prediction. A feature layer with a stride of 4 achieves a balance between semantics and detail, capturing medium-scale structural information. This feature layer, after passing through the prediction head of this layer, generates a coarse prediction. The feature layer with a stride of 1 reverts to pixel-level resolution, containing the finest texture and edge details. This feature layer, after being processed by the prediction head of this layer, generates a coarse prediction. A rough prediction , , As side outputs, these constitute three levels of prediction results, from coarse to fine, corresponding to the optimization stages of semantic dominance, structural recovery, and detail enhancement, respectively. These side outputs are input into the progressive refinement module, which achieves adaptive fusion of prediction results through a hierarchical self-guided mechanism. This allows different levels to focus on optimizing their respective most effective regions, guiding the network to generate refined predictions. Predictive mask.

[0070] The structure of progressively refined modules is as follows Figure 7 As shown, the coarse prediction produced by a feature layer with a step size of 8 No detailed processing, just regulations For the first Layer, the refined prediction mask of the previous layer After upsampling, a self-guiding mask is generated according to formula (7). ,in Indicates the x-coordinate of the current pixel. Represents the y-coordinate of the current pixel. When A value of 0 or 1 indicates that the predicted value for the current position is definite, corresponding to either the background or foreground, and no further refinement is needed from the current layer. Otherwise, it indicates that the position is in the transition region between the foreground and background, belonging to the uncertain part of the prediction, and the current layer needs to use lower-level features for more refined prediction. This yields the self-guiding mask. Then, according to formula (8), the current layer refines the prediction mask. Compared with the previous layer of refined prediction Selective fusion is performed. For unknown regions, the detailed features of the current layer are used for optimization, focusing on the prediction of unknown regions and avoiding semantic errors. For known regions, the high-confidence results of the previous layer are retained to ensure semantic consistency between different layers.

[0071]

[0072]

[0073] Step 2.5: Construct a weighted loss to enhance the model's decision-making ability at fuzzy boundaries;

[0074] The coarse-grained branch has a larger receptive field, enabling it to capture the overall contour and semantic information of foreground objects, determine the approximate range and overall structure of foreground objects in the image, and provide preliminary global semantic guidance for image matting. The fine-grained branch retains high-frequency details and structural information, enhancing the model's ability to handle complex edges. Based on the characteristics of the two branches, the features generated by the coarse-grained branch are refined... Mask prediction calculates training loss By weight Weighting; refining the generation of fine-grained branches. Mask prediction calculates training loss By weight Weighted average to obtain the total loss As shown in formula (9), the network is guided to learn global semantic information while paying attention to detailed information such as object edges.

[0075]

[0076] Step 3: Use a dual-branch video matting model to perform video sequence matting on consecutive image frames;

[0077] consecutive image frames The user-annotated first frame triad is input into the dual-branch video matting model. The dual-branch video matting model generates a refined first frame triad based on the first frame image and the first frame triad through the matting module. Mask prediction, then the first image frame, the first frame trisection, and the first frame refinement. The mask prediction and the next frame image are processed by the triangulation module to generate the next frame predicted triangulation. The next frame image and the predicted triangulation are then input into the matting module to generate a coarse matting image. The mask prediction is finally refined through the progressive refinement module. Mask prediction. The above process is used to iteratively process consecutive image frames until all frames are processed, thus achieving image matting of the video sequence.

[0078] To verify the effectiveness of the proposed dual-branch video matting model in capturing video frame details and improving temporal consistency, it was quantitatively compared with current mainstream video matting methods with auxiliary input on the VideoMatting108 dataset, and qualitative analysis was performed using visualization results. The experimental results are shown in Table 1.

[0079] Table 1. A dual-branch video matting model proposed in this invention.

[0080] Experimental results on the VideoMatting108 dataset

[0081] method MSE↓ SSDA↓ MAD↓ MESSDdt↓ dtSSD↓ OTVM 8.58 50.51 37.16 1.63 28.28 TCVOM 19.80 69.96 51.21 2.72 29.76 HSTSG 12.48 56.09 37.97 1.86 28.23 Ours 7.34 49.06 36.49 1.68 28.15

[0082] As shown in Table 1, except for the MESSDdt metric, the proposed dual-branch video matting model outperforms other algorithms on the VideoMatting108 dataset. This model improves the MAD metric by 0.67 compared to the OTVM method, indicating that the model's predictions are more accurate. The mask deviates less from the ground truth at the pixel level, resulting in more precise segmentation of foreground, background, and unknown regions. This allows for more accurate capture of object boundaries and details, leading to a better reproduction of realistic visual effects. While the squared terms of SSDA and MSE amplify significant errors, this model improves upon the OTVM method by 1.24 on MSE and 1.45 on SSDA, indicating improved prediction accuracy for complex regions, reduced extreme errors, and the achievement of high-quality image matting.

Claims

1. A method for dual-branch video keying, characterized in that, The implementation process of this method is as follows: Step 1: Obtain the original video sequence and the triangulation sequence. The first frame of the triangulation sequence is annotated by the user. Extract consecutive image frames from the original video sequence. ,in The number of consecutive image frames. This refers to the frame number of the image frame in the original video stream. ; Step 2: Construct a dual-branch video matting model; A dual-branch video matting model is constructed based on the U-Net network model, including a triangulation module, a matting module, and a progressive refinement module. The other triangulations in the triangulation sequence, except for the first frame triangulation, are predicted by the triangulation module. The matting module includes two branches, each of which includes an encoder module, a feature fusion module, and a decoder module. The dual-branch video matting model uses the first image frame and the user-annotated first-frame tri-image from the original video sequence as initial inputs. The stitched first image frame and the user-annotated first-frame tri-image are directly input to the matting module without passing through the tri-image propagation module. The matting module, based on prior information about unknown regions provided by the tri-image, extracts image features through the encoder module and performs information interaction and semantic enhancement through the feature fusion module. The image feature pyramid obtained by the feature fusion module is input to the decoder module for coarse image matting. Mask prediction generates three different resolution prediction results, which are then iteratively optimized by a progressive refinement module to finally obtain the refined result. Mask prediction; first frame image, first frame trisection, refinement The mask prediction is then input into the tri-image propagation module as memory information. The next image frame is input into the tri-image propagation module as query information. The tri-image propagation module integrates spatiotemporal information through the spatiotemporal memory module, calculates feature similarity, and obtains the next frame's predicted tri-image through the decoder module. The next image frame and the predicted tri-image are then refined by the matting module. Mask prediction, in this way The mask prediction process loops through consecutive image frames until all frames are processed. Step 3: Use a dual-branch video matting model to perform video sequence matting on consecutive image frames; The continuous image frames and the user-annotated first-frame triad are input into the dual-branch video matting model. The dual-branch video matting model generates a refined first-frame image based on the first-frame image frame and the first-frame triad using the matting module. Mask prediction, then the first image frame, the first frame trisection, and the first frame refinement. The mask prediction and the next frame image are processed by the triangulation module to generate the next frame predicted triangulation. The next frame image and the predicted triangulation are then input into the matting module to generate a coarse matting image. The mask prediction is finally refined through the progressive refinement module. Mask prediction; process consecutive image frames in a loop according to the above process until the process is complete, thus achieving image matting of the video sequence.

2. The method for dual-branch video matting according to claim 1, characterized in that, The coarse-grained branches of the matting network module constructed in step 2 include a coarse-branch encoder module, a feature fusion module, and a coarse-branch decoder module; The coarse-branch encoder module uses the Swing Transformer as the backbone network to construct hierarchical feature representations and generate feature maps at different scales; this module uses image frames... And predicting the three-part chart As input, where This indicates the current image frame number being processed. The input image is a 4×4 block segmented by the embedding layer. Each self-attention block serves as a sub-block of the coarse-branch encoder module. The self-attention block calculates local attention and performs cross-window interactions through a shift-window attention mechanism to obtain the image feature pyramid. ,in This represents the image feature map output by the coarse branch encoder module. Indicates the current floor number. This indicates the index of the feature map in the current layer; The attention-guided feature fusion module receives the image feature pyramid output by the encoder. The current level features and the previous level features are fused through the hybrid attention upsampling module, which retains the original semantic information while incorporating richer detailed features; The processing flow of the feature fusion module is shown in formulas (1) and (2), where, Indicates the first The first level Each feature map This represents the image feature map output by the coarse branch encoder module. Indicates the current floor number. This indicates the index of the feature map in the current layer. This indicates a hybrid attention upsampling operation. Indicates the first The first level The residual features output after the feature maps are processed by the hybrid attention upsampling operation; The fused features are reduced in number of channels by skip connections, compressing high-dimensional features to a suitable dimension and avoiding feature redundancy. The dimensionality-reduced features are then input into the decoder to restore the resolution layer by layer. , The coarse branch decoder consists of four residual upsampling blocks and three prediction heads, enabling efficient mapping from deep features to high-resolution transparency maps. Each sub-block introduces skip connections to preserve detail information, while the prediction heads predict feature maps at different scales, producing three different resolution side outputs for subsequent progressive refinement modules.

3. The method for dual-branch video matting according to claim 1, characterized in that, The fine-grained branching module of the matting network constructed in step 2 includes a fine-branch encoder module, a feature fusion module, and a fine-branch decoder module; The fine-branch encoder module uses the Swing Transformer as the backbone network to construct hierarchical feature representations and generate feature maps at different scales; this module uses image frames... And predicting the three-part chart As input, where This indicates the current image frame number being processed. The input image is segmented into 2×2 blocks after being processed by the embedding layer, allowing for more precise capture of subtle changes in the image and enhancing the model's ability to perceive image details and complex textures. Each self-attention block serves as a sub-block of the fine-branch encoder module. The self-attention block calculates local attention and performs cross-window interactions through a shift-window attention mechanism to obtain the image feature pyramid. ,in This represents the image feature map output by the fine-branch encoder module. Indicates the current floor number. This indicates the index of the feature map in the current layer; The attention-guided feature fusion module receives the image feature pyramid output by the encoder. The current level features and the previous level features are fused through the hybrid attention upsampling module, which retains the original semantic information while incorporating richer detailed features; feature The fusion process is shown in formulas (3) and (4), where, Indicates the first The first level Each feature map This represents the image feature map output by the fine-branch encoder module. Indicates the current floor number. This indicates the index of the feature map in the current layer. This indicates a hybrid attention upsampling operation. Indicates the first The first level The residual features output after the feature maps are processed by the hybrid attention upsampling operation; The fused features are reduced in number of channels by skip connections, compressing high-dimensional features to a suitable dimension and avoiding feature redundancy. The dimensionality-reduced features are then input into the decoder to restore the resolution layer by layer. , The fine-branch decoder consists of four residual upsampling blocks and three prediction heads, achieving efficient mapping from deep features to high-resolution transparency maps. Each sub-block introduces skip connections to preserve detail information, while the prediction heads predict feature maps at different scales, producing three different resolution side outputs for subsequent progressive refinement modules.

4. The method for dual-branch video matting according to claim 1, characterized in that, The weighted loss constructed in step 2 enhances the model's decision-making ability at fuzzy boundaries; The features of the coarse-grained branch have a larger receptive field, enabling them to capture the overall contour and semantic information of foreground objects, determine the approximate range and overall structure of foreground objects in the image, and provide preliminary global semantic guidance for image matting. The features of the fine-grained branch retain high-frequency details and structural information, enhancing the model's ability to handle complex edges. Based on the characteristics of the coarse-grained and fine-grained branches, the features generated by the coarse-grained branch are refined. Mask prediction calculates the training loss L1 using weights w1; fine-grained sub-scores are generated and refined. Mask prediction calculates training loss By weight Weighted average to obtain the total loss As shown in formula (5), the network is guided to learn global semantic information while paying attention to the details of the object's edge; 。

Images