Binocular video reflection removal method guided by depth information

Abstract

The invention provides a reflection-containing binocular video reflection removal method based on depth information guidance. The method comprises a transmission image depth estimation module and a reflection removal module. Wherein the transmission image depth estimation module performs depth perception on a binocular video stream frame by frame, decouples a mixed depth corresponding to the binocular video stream to obtain a real depth corresponding to a transmission layer and a pseudo depth corresponding to a reflection layer, and adopts a real depth corresponding to a transmission image to guide a reflection removal network to realize effective reflection removal. The reflection removal module realizes fusion and enhancement of cross-view features, depth guide features and cross-frame features by implanting a feature fusion and enhancement module with a uniform structure, and a gating controller is designed for a CVEM module, a DGEM module and a CFEM module to control the feature relationship exploration range of three different tasks. And a good effect of reflection removal of the given reflection-containing video stream is realized.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a binocular video reflection removal method guided by depth information. Background Art

[0002] When capturing images or shooting videos through scenes blocked by glass, obvious reflection artifacts are usually observed, which leads to cluttered, blurred, missing details, and degraded image quality, seriously affecting people's understanding of the image or video content. Reflection artifacts in images or videos will have a great negative impact on downstream tasks, such as image and video detection, tracking, segmentation, classification and other computer vision tasks. Research on reflection removal can promote the development and application of autonomous driving, video surveillance, intelligent photography and special scene shooting.

[0003] With the development of computer vision algorithms, removing reflections from a single image has become a common approach. However, removing reflections from a single image is challenging, and there are countless possible solutions to this problem without additional guidance. Therefore, introducing auxiliary information to guide reflection removal under such challenging conditions is an effective approach. Recent studies have found that depth information is an effective guiding information for removing reflections. For example, active near-infrared sensors can be used to remove reflections by obtaining the depth of the transmitted scene. However, this approach requires specific sensors and a specific angle range between the shooting direction and the glass, both of which limit its application in more general scenes. For more widely used commercial monocular cameras, a common method of perceiving the depth of transmitted and reflected scenes is to adjust the focal length to keep the transmitted scene within the field of view, thereby blurring the edges of the reflected scene, which is conducive to reflection identification and removal. For example, using the depth of field estimated from a single image to identify the edge of the reflection provides a feasible solution for removing reflections. However, when the mixed image contains complex textures, the robustness of the single-image depth estimation model decreases, and the information provided for reflection removal is limited. In order to perceive depth more robustly, a strategy of capturing a pair of images with different focus settings can be used for depth perception. However, this strategy is limited to static scenes. In more general dynamic scenes, it is very challenging to adjust the focus to keep up with the motion of objects within the scene. Therefore, the removal of reflections relies on the robustness of the depth estimation.

[0004] Compared with monocular cameras, binocular cameras can capture a pair of two-view images with known binocular camera parameters at the same time. Using this feature, the two-view images can be aligned at the pixel level and the reflection invariant flow can be estimated for the two-view views. However, these methods ignore the perception of scene depth. For human vision, rich prior knowledge enables us to intuitively distinguish the real depth produced by the transmission scene and the pseudo depth produced by the reflection scene. Inspired by the human binocular vision system, we can design a deep learning model through a large amount of training on the binocular video stream data to obtain prior knowledge that can effectively distinguish the real depth from the pseudo depth. This learned knowledge can then be used to remove reflections. Therefore, this patent proposes an end-to-end depth-guided reflection removal network for binocular video reflection removal. The network includes a transmission map depth estimation module and a reflection removal module. In addition, the binocular video contains intrinsic cross-view and cross-frame information. Therefore, this patent uses depth guidance, supplemented by cross-view and cross-frame feature fusion and enhancement, to achieve effective reflection removal. To fully exploit this rich guidance, we design a unified feature fusion and enhancement module to explore the correlations between local and global features, equipped with a specially designed gating controller to control the feature relationship exploration scope of the three different tasks according to their feature points. Summary of the invention

[0005] An embodiment of the present invention provides a binocular video reflection removal method guided by depth information, which is used to solve the technical problems existing in the prior art.

[0006] In order to achieve the above object, the present invention adopts the following technical scheme.

[0007] A binocular video reflection removal method guided by depth information, comprising:

[0008] Based on the left and right views of multiple reflection video frames, the transmission depth maps of the multiple reflection video frames are processed by the transmission map depth estimation module of the network model to obtain the transmission depth maps of the multiple reflection video frames; the multiple video frames include the current video frame, the previous video frame and the subsequent video frame; the transmission depth map includes the transmission depth left view, the transmission depth right view and the transmission depth left and right views;

[0009] Based on the left and right views of multiple reflection video frames and the corresponding transmission depth maps, feature fusion and reflection removal processing are performed through the reflection removal module of the network model to obtain the transmission map video frame;

[0010] The reflection removal module is based on an encoder-decoder structure, including a first encoder, a second encoder, a third encoder, and a fourth encoder arranged in parallel with each other, and a decoder, and also has a cross-view feature enhancement module, a depth-guided feature enhancement module, and a cross-frame feature enhancement module;

[0011] The first encoder and the second encoder are used to perform step-by-step feature extraction on the left view and the right view of the current video frame respectively; the third encoder is used to perform step-by-step feature extraction on the transmission depth left and right views of the current video frame; the fourth encoder is used to perform step-by-step feature extraction on the left and right views of multiple reflection video frames and the corresponding transmission depth maps;

[0012] The cross-view feature enhancement module performs a left-right view feature complementation operation on the outputs of the first encoder and the second encoder to obtain the left-right feature maps of the enhanced reflection video frame, and aligns the features of the left-right feature maps of the enhanced current video frame with the transmission depth map at the pixel level; the depth-guided feature enhancement module performs depth information guidance and feature integration operations on the outputs of the cross-view feature enhancement module and the third encoder to obtain the feature map after depth-guided feature enhancement; the inter-frame feature enhancement module is used to perform a fusion operation on the outputs of the depth-guided feature enhancement module and the fourth encoder to obtain the feature map after inter-frame feature enhancement.

[0013] Preferably, based on the left and right views of the multiple reflection video frames, the process of obtaining the transmission depth map of the multiple reflection video frames by processing through the transmission map depth estimation module includes:

[0014] Video sequences captured by a stereo camera with multiple reflection video frames The calculation formula of the depth estimation module through the transmission map

[0015]

[0016] Get the transmission depth map of the reflected video frame Where i∈[0,N-1] represents the index of the video frame.

[0017] Preferably: the first encoder uses the left view of the current video frame As input, output the left view feature map f of the current video frame l m ; The second encoder uses the right view of the current video frame As input, output the right view feature map of the current video frame The third encoder uses the transmission depth of the left and right views corresponding to the left and right views of the current video frame As input, output the transmission depth feature map of the current video frame The fourth encoder converts the previous video frame, the current video frame and the subsequent video frame And its corresponding left and right view transmission map depth As input, output inter-frame feature map

[0018] The first encoder, the second encoder, the third encoder and the fourth encoder each have three feature extraction blocks; the b-th feature extraction block among the three feature extraction blocks consists of b convolutional layers and b CBAMs (b=1, 2, 3);

[0019] The processing of the cross-view feature enhancement module specifically includes:

[0020] Set the left view feature map f of the current video frame input l m and right view feature map The feature vector for each position (x,y) is and Where c represents the feature dimension;

[0021] The left view feature map f of the current video frame l m and right view feature map Perform global pooling operation to obtain the global feature vector and

[0022] Pass-through

[0023]

[0024] For the eigenvector f l m (x,y) and and the global eigenvector and Combine to obtain point feature group In the formula, cat(·) represents the feature combination operation;

[0025] Pass-through

[0026]

[0027] Perform feature integration; where H j (·)(j∈[1,J]) means that each point feature group is processed separately through the multi-head attention mechanism

[0028] Constructed to build The similarity matrix of the relationship between the four components in And input into the first gate controller to control the feature extraction direction to obtain an updated feature map and

[0029] The updated feature map and Perform fusion processing to obtain the fused feature map

[0030] The processing process of the depth-guided feature enhancement module specifically includes:

[0031] Set the input fused feature map And the left and right feature maps of the current video frame The feature vector for each position (x,y) is and Where c represents the feature dimension;

[0032] For the fused feature map And left and right feature maps Perform global pooling operation to obtain the global feature vector and

[0033] Pass-through

[0034]

[0035] For the feature vector and and the global eigenvector and Combine to obtain point feature group In the formula, cat(·) represents the feature combination operation;

[0036] Pass-through

[0037]

[0038] Perform feature integration; where H j (·)(j∈[1,J]) means that each point feature group is processed separately through the multi-head attention mechanism

[0039] Constructed to build The similarity matrix of the relationship between the four components in And input into the second gate controller to control the feature extraction direction to obtain an updated feature map and

[0040] The updated feature map and Perform fusion processing to obtain the fused feature map

[0041] The processing process of the cross-frame feature enhancement module specifically includes:

[0042] Set the input fused feature map and inter-frame feature maps The feature vector for each position (x,y) is and Where c represents the feature dimension;

[0043] For the fused feature map and inter-frame feature maps Perform global pooling operation to obtain the global feature vector and

[0044] Pass-through

[0045]

[0046] For the feature vector and and the global eigenvector and Combine to obtain point feature group In the formula, cat(·) represents the feature combination operation;

[0047] Pass-through

[0048]

[0049] Perform feature integration; where H j (·)(j∈[1,J]) means that each point feature group is processed separately through the multi-head attention mechanism

[0050] Constructed to build The similarity matrix of the relationship between multiple components in And input into the third gate controller to control the feature extraction direction to obtain an updated feature map and

[0051] The updated feature map and Perform fusion processing to obtain the fused feature map

[0052] Preferably, the processing of the first gate controller includes:

[0053] Pass-through

[0054]

[0055] Control the feature extraction direction to obtain an updated feature map and In the formula, G1(·) represents the first gate controller, M2 is Matrix of

[0056] The processing of the second gate controller includes:

[0057] Pass-through

[0058]

[0059] Control the feature extraction direction to obtain an updated feature map and Where G2(·) represents the second gate controller, M1 is an anti-diagonal matrix, and each anti-diagonal element value of the anti-diagonal matrix is ​​-100;

[0060] The processing of the first gate controller includes:

[0061] Pass-through

[0062]

[0063] Control the feature extraction direction to obtain an updated feature map and G3(·) represents the third gate controller, M2 is The matrix of .

[0064] Preferably, the network model also has a loss function, which is expressed by

[0065] L=ω1L1+ω2L ssim +ω3L rec +ω4L mse +ω5L mp

[0066] Calculate the overall loss to train the network model; where L1, L ssim ,L rec ,L mse ,L mp They represent respectively: L1 pixel loss, SSIM structure loss, reconstruction loss, mean square error loss, and perceptual loss; ω1=0.4, ω2=0.4, ω3=0.4, ω4=0.4, ω5=0.1.

[0067] It can be seen from the technical solutions provided by the above embodiments of the present invention that the present invention provides a method for removing reflections from binocular videos containing reflections based on depth information guidance, and the method includes a transmission map depth estimation module and a reflection removal module. The transmission map depth estimation module performs depth perception on the binocular video stream frame by frame, decouples the mixed depth corresponding to the binocular video stream to obtain the real depth corresponding to the transmission layer and the pseudo depth corresponding to the reflection layer, and uses the real depth corresponding to the transmission map to guide the reflection removal network to achieve effective reflection removal. The reflection removal module realizes cross-view features, depth feature guidance and cross-frame feature fusion and enhancement by implanting a feature fusion and enhancement module with a unified structure, and controls the feature relationship exploration range of three different tasks by designing a gate controller for the CVEM module, the DGEM module and the CFEM module. Finally, a good effect of removing reflections from a given video stream containing reflections is achieved. The algorithm proposed in the invention shows better PSNR, SSIM, NCC, and LMSE indicators in the comparison of the existing single-image and multi-image reflection removal algorithms, and is superior to the comparison algorithm in the visualization effect after reflection removal.

[0068] Additional aspects and advantages of the present invention will be given in part in the following description, which will become obvious from the following description, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0070] Figure 1 A processing flow chart of a binocular video reflection removal method guided by depth information provided by the present invention;

[0071] Figure 2 A flowchart of a preferred embodiment of a binocular video reflection removal method guided by depth information provided by the present invention;

[0072] Figure 3 A schematic diagram of the interactive flow of three feature enhancement modules of a binocular video reflection removal method guided by depth information provided by the present invention;

[0073] Figure 4 A comparison chart of the effects of a binocular video reflection removal method guided by depth information provided by the present invention and an algorithm in the prior art. DETAILED DESCRIPTION

[0074] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0075] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or coupling. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.

[0076] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0077] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.

[0078] The present invention provides a binocular video reflection removal method guided by depth information, which is used to solve the following technical problems found by the applicant in practice:

[0079] (1) Previous reflection removal algorithms based on single images, multiple images, and monocular videos ignored the perception principle of human binocular vision for scenes containing reflections. We propose to use the currently widely used binocular camera to capture dual-view videos to model the ability of the human eye to perceive scene depth, thereby providing effective depth guidance information for reflection removal. (2) Previous binocular video reflection removal algorithms ignored the perception of depth information in the scene. We propose to decouple the mixed depth estimated by the binocular video to obtain the true depth corresponding to the transmission layer and the pseudo depth corresponding to the reflection layer, and use the true depth corresponding to the transmission map to guide the reflection removal network to achieve effective reflection removal. (3) For the existing binocular video reflection removal algorithm model, there is a lack of detailed design of feature fusion between inter-view features, inter-frame features, and guidance information features. Designing an effective feature fusion method is crucial to the integration of cross-view and cross-frame features, as well as the effectiveness of depth feature guidance.

[0080] In view of this, the present invention adopts the currently widely used binocular camera to perform scene depth perception through the captured left and right perspective videos, thereby providing effective depth guidance information for reflection removal. The mixed depth estimated from the binocular video is used to obtain the real depth corresponding to the transmission layer and the pseudo depth corresponding to the reflection layer, and the real depth corresponding to the transmission map is used to guide the reflection removal network to achieve effective reflection removal. An effective feature enhancement method is designed in the reflection removal module so that the cross-perspective, depth-guided features and cross-frame features are fully integrated and enhanced to achieve a better binocular video reflection removal effect.

[0081] See also Figure 1 The present invention provides a binocular video reflection removal method guided by depth information, comprising:

[0082] Based on the left and right views of multiple reflection video frames, the transmission depth maps of the multiple reflection video frames are processed by the transmission map depth estimation module of the network model to obtain the transmission depth maps of the multiple reflection video frames; the multiple video frames include the current video frame, the previous video frame and the subsequent video frame; the transmission depth map includes the transmission depth left view, the transmission depth right view and the transmission depth left and right views;

[0083] Based on the left and right views of multiple reflection video frames and the corresponding transmission depth maps, feature fusion and reflection removal processing are performed through the reflection removal module of the network model to obtain the transmission map video frame;

[0084] The reflection removal module is based on an encoder-decoder structure, including a first encoder, a second encoder, a third encoder, and a fourth encoder arranged in parallel with each other, and a decoder, and also has a cross-view feature enhancement module, a depth-guided feature enhancement module, and a cross-frame feature enhancement module;

[0085] The first encoder and the second encoder are used to perform step-by-step feature extraction on the left view and the right view of the current video frame respectively; the third encoder is used to perform step-by-step feature extraction on the transmission depth left and right views of the current video frame; the fourth encoder is used to perform step-by-step feature extraction on the left and right views of multiple reflection video frames and the corresponding transmission depth maps;

[0086] The cross-view feature enhancement module performs a left-right view feature complementation operation on the outputs of the first encoder and the second encoder to obtain the left-right feature maps of the enhanced reflection video frame, and aligns the features of the left-right feature maps of the enhanced current video frame with the transmission depth map at the pixel level; the depth-guided feature enhancement module performs depth information guidance and feature integration operations on the outputs of the cross-view feature enhancement module and the third encoder to obtain the feature map after depth-guided feature enhancement; the inter-frame feature enhancement module is used to perform a fusion operation on the outputs of the depth-guided feature enhancement module and the fourth encoder to obtain the feature map after inter-frame feature enhancement.

[0087] The above steps are performed multiple times and all the transmission image video frames are fused to obtain a video with reflection removed.

[0088] The reflection removal network designed by the present invention mainly includes a transmission map depth estimation module and a reflection removal module. The network is based on CNN and uses an iterative cycle to remove reflections frame by frame. In a preferred embodiment, the specific execution process can be:

[0089] Given a video sequence captured by a stereo camera where i∈[0,n-1] represents the index of the video frame. We design a depth estimation module d(·) to achieve depth perception and obtain a transmission depth map from the mixed depth. This module takes the left and right views of the captured video frame with reflection as input and generates the transmission depth map of the left and right views of the video frame. The module can be expressed as: The remaining part of the network (denoted as the reflection removal module T(·)) is used to perform feature fusion and reflection removal. The first N groups of left and right view video frames, the current left and right view video frame, and the subsequent N groups of left and right view video frames ( A total of 2N+1 groups) and their corresponding left and right view transmission image depth As input, generate the final transmission map The process can be described as: In the feature extraction stage of the reflection removal module T(·), the present invention designs three key modules at different levels of feature scales, which interact and enhance: cross-view features, depth-guided features, and cross-frame features. First, a cross-view feature enhancement module (CVEM) is designed to fuse the left and right view features of the current video frame. Then a depth-guided feature enhancement module (DGEM) is designed to fuse the features fused by the CVEM module with the depth features, and give play to the guiding role of the transmission map in reflection removal. Finally, a cross-frame feature enhancement module (CFEM) is designed to fuse the feature map after depth guidance with the feature map of the 2N+1 group of video frame sequences, making full use of the inter-frame information of the binocular video. Through the effects of the above series of feature enhancement modules, the CVEM module makes the left and right view features complement each other, and at the same time aligns the left and right view features and the depth map at the pixel level; the DGEM module guides the depth information and integrates the features; the CFEM module then fuses the rich inter-frame information of the video, and is conducive to the fluency of the video. In the present invention, the above three feature interaction modules are summarized into a unified feature interaction structure and matched with different gate controllers.

[0090] The transmission image depth estimation module D(·) of the reflection removal network provided by the present invention is designed based on the UNet structure. In a feasible embodiment, the module takes the left and right views of the reflection video frame as input and performs depth estimation from coarse to fine. First, the original Figure 1 The module uses two residual blocks, four atrous convolutional layers, and two convolutional block attention modules (CBAMs) to expand the receptive field and enrich the diversity of the intermediate layer convolutions.

[0091] The reflection removal module T(·) in this embodiment is designed based on an encoder-decoder structure and consists of four encoders and a decoder. The first encoder uses the left view of the current frame of the binocular video As input, output the left view feature map f of the current video frame l m ; The second encoder uses the right view of the current frame of the binocular video As input, output the right view feature map of the current video frame The third encoder takes the left and right views of the current frame of the binocular video as input to the transmission depth left and right views generated by D(·) As input, output the transmission depth feature map of the current video frame The fourth encoder converts the first N groups of left and right view video frames, the current left and right view video frame, and the subsequent N groups of left and right view video frames. And its corresponding transmission depth left and right view As input, output inter-frame feature map The four encoders adopt the same structure to extract features step by step. Each encoder uses three feature extraction blocks, where the bth feature extraction block consists of b convolutional layers and b CBAMs (b=1, 2, 3). Between the four encoders, the CVEM module, DGEM module and CFEM module of the unified structure designed by the present invention with different gate controllers are implanted to perform cross-view, depth guidance and cross-frame feature interaction and feature enhancement respectively. The decoder converts the fused and enhanced feature map into As input, 4 dilated convolution blocks, 2 CBAM blocks, 2 sets of deconvolution and convolution blocks are used to generate the final transmission map video frame.

[0092] In this embodiment, the CVEM module, the DGEM module, and the CFEM module are connected in a cascaded manner between the four encoder branches of the reflection removal module. Except for the difference in the gate controller, their structures are similar. The mth CVEM takes the left and right view feature maps f at the current level. l m and As input, we get 3 outputs: Then, the enhanced features generated by CVEM With deep features Combining, we get: Then, CFEM uses the depth-enhanced feature map obtained from DGEM and inter-frame feature maps Combine to get The updated feature map will be processed by the convolution block and CBAM, and then the next feature enhancement will be performed in a higher dimension. The feature map processing (update) process of these three modules at the mth level can be described as:

[0093]

[0094] Among them, CV(·), DG(·), and CF(·) represent the CVEM module, DGEM module, and CFEM module, respectively.

[0095] The following takes DEGM as an example to introduce the specific operations within each module. and The eigenvector of each position (x, y) on is recorded as Where c represents the feature dimension. First, the feature map and Perform global average pooling (GAP) to obtain the global feature vector and For each position (x, y), this embodiment converts the feature vector and Combined together, a point feature group is formed:

[0096]

[0097] Where cat(·) represents the feature combination operation. Then, a multi-head attention mechanism is adopted, using H j (·)(j∈[1,J]) processes each point feature group separately Then they are combined and a linear layer is used to integrate the features. This process can be expressed as:

[0098]

[0099] In each H j (·), this embodiment constructs a similarity matrix To establish the relationship between the four components, the four components are The 4 elements in cat(·) Then, the similarity matrix The input gate controller controls the direction of various feature correlation extraction. This embodiment will introduce the specific design of the gate controller of CVEM, DGEM and CFEM in the next section. Under the action of the gate controller, the feature fusion and enhancement module of the unified structure can effectively learn the correlation between the four components and dynamically adjust the feature vector group through matrix multiplication. Then, the adjusted feature vectors in the vector group are redistributed to the corresponding positions to obtain the updated feature map: In order to obtain fused features and provide global guidance for subsequent modules, this embodiment combines the two obtained feature maps and uses 1×1 convolution to integrate their features to obtain a fused feature map.

[0100] Similarly, the processing of CVEM is as follows:

[0101] Set the left view feature map f of the current video frame input l m and right view feature map The feature vector for each position (x,y) is and Where c represents the feature dimension;

[0102] The left view feature map f of the current video frame l m and right view feature map Perform global pooling operation to obtain the global feature vector and

[0103] Pass-through

[0104]

[0105] For the eigenvector f l m (x,y) and and the global eigenvector and Combine to obtain point feature group In the formula, cat(·) represents the feature combination operation;

[0106] Pass-through

[0107]

[0108] Perform feature integration; where H j (·)(j∈[1,J]) means that each point feature group is processed separately through the multi-head attention mechanism

[0109] Constructs the four components Similarity matrix of the relationship between And input into the first gate controller to control the feature extraction direction to obtain an updated feature map and

[0110] The updated feature map and Perform fusion processing to obtain the fused feature map

[0111] Similarly, the processing process of CFEM is as follows:

[0112] Set the input fused feature map and inter-frame feature maps The feature vector for each position (x,y) is and Where c represents the feature dimension;

[0113] For the fused feature map and inter-frame feature maps Perform global pooling operation to obtain the global feature vector and

[0114] Pass-through

[0115]

[0116] For the feature vector and and the global eigenvector and Combine to obtain point feature group In the formula, cat(·) represents the feature combination operation;

[0117] Pass-through

[0118]

[0119] Perform feature integration; where H j (·)(j∈[1,J]) means that each point feature group is processed separately through the multi-head attention mechanism

[0120] Constructs the four components Similarity matrix of the relationship between And input into the third gate controller to control the feature extraction direction to obtain an updated feature map and

[0121] The updated feature map and Perform fusion processing to obtain the fused feature map

[0122] In this embodiment, three unified feature fusion and enhancement modules: CVEM module, DGEM module and CFEM module are used to establish relationships between four encoders in a cascaded manner. The gate controller is designed to control the feature relationship exploration range of three different tasks. For the second gate controller of DGEM, the left and right views of the video frame used to extract features and their transmission map depth are point-to-point aligned, so the second gate controller of this module uses the jth similarity matrix The sum of the inverse angle matrix M1 is used as input, and a softmax layer is used to constrain the inverse angle value to be close to 0. The gate controller G1(·) applied in DGEM suppresses the interaction between the global information of one modality feature map and the local information of another modality feature map. The gate controller can be described as:

[0123] where M1 is an anti-diagonal matrix with each anti-diagonal element value equal to -100. In contrast, the input paired feature maps of CVEM and CFEM are not aligned. It is meaningful to explore the local and global correlations in the two feature maps. Specifically, the gate controller G2(·) applied to the CVEM and CFEM modules is based on the j-th similarity matrix Add a matrix M2 as input and activate it through the softmax layer. The process can be expressed as:

[0124] Where M2 is

[0126] In the preferred embodiment provided by the present invention, the designed model is trained by introducing reconstruction loss, pixel loss, structure loss, perceptual loss and adversarial loss to optimize the performance of the network. The overall loss function used in the training network of the present invention is defined as follows:

[0127] L=ω1L1+ω2L ssim +ω3L rec +ω4L mse +ω5L mp

[0128] Among them, L1,L ssim ,L rec ,L mse ,L mp They respectively represent: L1 pixel loss, SSIM (Structural Similarity) structure loss, reconstruction loss, mean square error loss, and perceptual loss; ω1=0.4, ω2=0.4, ω3=0.4, ω4=0.4, ω5=0.1.

[0129] Attached Figure 4 Two visualization examples of the present invention are given. The first column is a frame image of the left view of the binocular video, the 2nd to 6th columns are the visualization results of the comparison algorithm, the 7th and 9th columns are the visualization results of the transmission map and transmission depth map of the present invention, and the 8th and 10th columns are the real result reference images corresponding to the examples.

[0130] In summary, the present invention provides a method for removing reflections from binocular videos containing reflections based on depth information guidance, the method comprising a transmission map depth estimation module and a reflection removal module. The transmission map depth estimation module performs depth perception on the binocular video stream frame by frame, decouples the mixed depth corresponding to the binocular video stream to obtain the real depth corresponding to the transmission layer and the pseudo depth corresponding to the reflection layer, and uses the real depth corresponding to the transmission map to guide the reflection removal network to achieve effective reflection removal. The reflection removal module realizes cross-view features, depth feature guidance and cross-frame feature fusion and enhancement by implanting a feature fusion and enhancement module with a unified structure, and controls the feature relationship exploration range of three different tasks by designing a gate controller for the CVEM module, the DGEM module and the CFEM module. Finally, a good effect of removing reflections from a given video stream containing reflections is achieved. The algorithm proposed in the invention shows better PSNR, SSIM, NCC, and LMSE indicators in the comparison of the existing single-image and multi-image reflection removal algorithms, and is superior to the comparison algorithm in the visualization effect after reflection removal.

[0131] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0132] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention or certain parts of the embodiments.

[0133] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0134] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A binocular video reflection removal method guided by depth information, characterized in that: include: Based on the left and right views of multiple reflection video frames, the transmission depth maps of the multiple reflection video frames are processed by the transmission map depth estimation module of the network model to obtain the transmission depth maps of the multiple reflection video frames; the multiple video frames include the current video frame, the previous video frame and the subsequent video frame; the transmission depth map includes the transmission depth left view, the transmission depth right view and the transmission depth left and right views; Based on the left and right views of the multiple reflection video frames and the corresponding transmission depth maps, feature fusion and reflection removal processing are performed through the reflection removal module of the network model to obtain a transmission map video frame; The reflection removal module is based on an encoder-decoder structure, including a first encoder, a second encoder, a third encoder, and a fourth encoder arranged in parallel with each other, and a decoder, and also has a cross-view feature enhancement module, a depth-guided feature enhancement module and a cross-frame feature enhancement module; The first encoder and the second encoder are used to perform step-by-step feature extraction on the left view and the right view of the current video frame respectively; the third encoder is used to perform step-by-step feature extraction on the transmission depth left and right views of the current video frame; the fourth encoder is used to perform step-by-step feature extraction on the left and right views of the multiple reflection video frames and the corresponding transmission depth maps; The cross-view feature enhancement module performs a left-right view feature complementation operation on the outputs of the first encoder and the second encoder to obtain an enhanced left-right feature map of the reflected video frame, and aligns the features of the enhanced left-right feature map of the current video frame with the transmission depth map at the pixel level; the depth-guided feature enhancement module performs depth information guidance and feature integration operations on the outputs of the cross-view feature enhancement module and the third encoder to obtain a feature map after depth-guided feature enhancement; the inter-frame feature enhancement module is used to perform a fusion operation on the outputs of the depth-guided feature enhancement module and the fourth encoder to obtain a feature map after inter-frame feature enhancement.

2. The method according to claim 1, characterized in that The process of obtaining the transmission depth map of the multiple reflection video frames by processing the left and right views of the multiple reflection video frames through the transmission map depth estimation module includes: Video sequences captured by a stereo camera with multiple reflection video frames The calculation formula of the transmission map depth estimation module is Get the transmission depth map of the reflected video frame Where i∈[0,N-1] represents the index of the video frame.

3. The method according to claim 2, characterized in that: The first encoder uses the left view of the current video frame As input, output the left view feature map f of the current video frame l m ; The second encoder uses the right view of the current video frame As input, output the right view feature map of the current video frame The third encoder uses the transmission depth left and right views corresponding to the left and right views of the current video frame As input, output the transmission depth feature map of the current video frame The fourth encoder converts the previous video frame, the current video frame and the subsequent video frame And its corresponding left and right view transmission map depth As input, output inter-frame feature map The first encoder, the second encoder, the third encoder and the fourth encoder each have three feature extraction blocks; the bth feature extraction block among the three feature extraction blocks is composed of b convolutional layers and b CBAMs (b=1, 2, 3); The processing process of the cross-view feature enhancement module specifically includes: Set the left view feature map of the current video frame input and right view feature map The feature vector for each position (x,y) is and Where c represents the feature dimension; Left view feature map of the current video frame and right view feature map Perform global pooling operation to obtain the global feature vector and Pass-through For the eigenvector f l m (x,y) and and the global eigenvector and Combine to obtain point feature group In the formula, cat(·) represents the feature combination operation; Pass-through Perform feature integration; where H j (·)(j∈[1,J]) means that each point feature group is processed separately through the multi-head attention mechanism Constructed to build The similarity matrix of the relationship between the four components in And input into the first gate controller to control the feature extraction direction to obtain an updated feature map and The updated feature map and Perform fusion processing to obtain the fused feature map The processing process of the depth-guided feature enhancement module specifically includes: Set the input fused feature map And the left and right feature maps of the current video frame The feature vector for each position (x,y) is and Where c represents the feature dimension; For the fused feature map And left and right feature maps Perform global pooling operation to obtain the global feature vector and Pass-through For the feature vector and and the global eigenvector and Combine to obtain point feature group In the formula, cat(·) represents the feature combination operation; Pass-through Perform feature integration; where H j (·)(j∈[1,J]) means that each point feature group is processed separately through the multi-head attention mechanism Constructed to build The similarity matrix of the relationship between the four components in And input into the second gate controller to control the feature extraction direction to obtain an updated feature map and The updated feature map and Perform fusion processing to obtain the fused feature map The processing process of the cross-frame feature enhancement module specifically includes: Set the input fused feature map and inter-frame feature maps The feature vector for each position (x,y) is and Where c represents the feature dimension; For the fused feature map and inter-frame feature maps Perform global pooling operation to obtain the global feature vector and Pass-through For the feature vector and and the global eigenvector and Combine to obtain point feature group In the formula, cat(·) represents the feature combination operation; Pass-through Perform feature integration; where H j (·)(j∈[1,J]) means that each point feature group is processed separately through the multi-head attention mechanism Constructed to build The similarity matrix of the relationship between the four components in And input into the third gate controller to control the feature extraction direction to obtain an updated feature map and The updated feature map and Perform fusion processing to obtain the fused feature map 4. The method according to claim 3, characterized in that The processing process of the first gate controller includes: Pass-through Control the feature extraction direction to obtain an updated feature map and In the formula, G1(·) represents the first gate controller, M2 is Matrix of The processing process of the second gate controller includes: Pass-through Control the feature extraction direction to obtain an updated feature map and Where G2(·) represents the second gate controller, M1 is an anti-diagonal matrix, and each anti-diagonal element value of the anti-diagonal matrix is ​​-100; The processing process of the first gate controller includes: Pass-through Control the feature extraction direction to obtain an updated feature map and G3(·) represents the third gate controller, M2 is The matrix of .

5. The method according to claim 4, characterized in that The network model also has a loss function, which is expressed by L=ω1L1+ω2L ssim +ω3L rec +ω4L mse +ω5L mp Calculate the overall loss to train the network model; where L1,L ssim ,L rec ,L mse ,L mp They represent respectively: L1 pixel loss, SSIM structure loss, reconstruction loss, mean square error loss, and perceptual loss; ω1=0.4, ω2=0.4, ω3=0.4, ω4=0.4, ω5=0.1.

Images