Single image anti-reflective method and system based on binocular vision

Abstract

The present application relates to the technical field of computer vision, and discloses a single-image anti-reflection method and system based on binocular vision to reduce the parallax error of subsequent processing. The method comprises: constructing a data set, constructing an anti-reflection network structure, training and verifying, and deploying the trained anti-reflection network structure for testing; wherein the data set is composed of a series of image pairs collected synchronously by binocular cameras but not aligned, and in any image pair, the image without reflection is labeled, and the single image with reflection is marked; the anti-reflection network structure comprises: a primary feature extraction architecture, an encoder-decoder architecture based on a series of GAFBlock for depth processing to generate an initial prediction image without reflection, and a discriminator network architecture for discriminating the quality of the initial prediction image compared with the image without reflection in the image pair, and then combining the discrimination result with an attention mechanism for distinguishing the reflection area and the background area to generate a final prediction image without reflection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a single-image defogging method and system based on binocular vision. BACKGROUND

[0002] In the field of computer vision, image defogging technology is a core challenge to improve the reliability of vision systems in scenarios such as autonomous driving, security monitoring, and medical imaging. Reflection artifacts caused by smooth surfaces such as glass, mirrors, and water can severely block background information, leading to a decrease in the accuracy of tasks such as target detection and feature extraction. For example, in autonomous driving, the road preview system relies on non-reflective images to obtain environmental details to ensure driving safety. However, window reflections or road reflections often interfere with the feature recognition of the environment perception algorithm, directly threatening the accuracy of autonomous driving decisions.

[0003] In recent years, deep learning methods based on convolutional neural networks (CNN) have made progress in single-image defogging, but their training is highly dependent on strictly aligned image pairs (i.e., image pairs that differ only in the reflection layer and have completely aligned content). This paradigm exposes significant shortcomings in binocular vision scenarios: binocular imaging systems are difficult to meet the stringent requirements of traditional training for image alignment due to natural angle deviations, geometric distortions, and pixel-level misalignments. Existing algorithms lack the ability to process non-aligned data and cannot effectively separate the reflection layer and the background layer when processing binocular images, often resulting in issues such as detail loss, artifact residue, or structural distortion in the defogged images.

[0004] The processing difficulties of traditional methods and mainstream deep learning models for non-aligned images are essentially due to their modeling logic relying on pixel-level alignment. When the input images have angle differences or geometric distortions, the spatial correspondence between the reflection layer and the background layer is disrupted, making it difficult for the network to accurately distinguish between the two types of information. Although non-aligned images (such as handheld shooting in dynamic environments and multi-sensor collaborative imaging) are common in real-world scenarios, current research on defogging for this type of data is still in its infancy, with significant technical gaps. SUMMARY

[0005] The present application aims to disclose a single-image defogging method and system based on binocular vision to reduce parallax errors in subsequent processing.

[0006] To achieve the above purpose, the single image defogging method based on binocular vision of the application comprises the following steps: constructing a data set, dividing the data set into a training set and a validation set, constructing, training and validating a defogging network structure, and deploying the trained defogging network structure for testing; wherein the data set is composed of a series of image pairs synchronously collected by binocular cameras but not aligned, and in any of the image pairs, the image without defogging is labeled, and the single image with defogging is marked; the defogging network structure comprises:

[0007] a primary feature extraction architecture for feature extraction on the input image with defogging, and then the extracted features of each layer are adjusted to the same size as the input image by bilinear interpolation, and then spliced into a super column vector after being spliced with the input image with defogging, and then input into an encoder-decoder architecture;

[0008] the encoder-decoder architecture for deep processing based on a series of GAFBlock in the process of upsampling and downsampling to generate an initial predicted image without defogging; wherein each GAFBlock first performs layer normalization processing on the input features, then performs channel mapping on the normalized features through a 1x1 convolution, then extracts local features through a 3x3 depth separable convolution, then adjusts the flow of feature information through a gating mechanism realized by channel splitting and element-wise multiplication operation, and then performs SCA processing on the features processed by the gating mechanism, and then outputs the features adjusted by SCA through a 1x1 convolution for channel integration and residual connection with the input features;

[0009] a discriminator network architecture for discriminating the quality of the initial predicted image compared with the image without defogging in the image pair, and then generating a final predicted image without defogging by combining the discrimination result with an attention mechanism for distinguishing the reflection area and the background area;

[0010] wherein in the training process of the defogging network structure, the weighted sum of the alignment invariant loss and the generative adversarial loss of the discriminator network architecture is taken as the total loss for iterative adjustment of the model parameters, and the alignment invariant loss is the feature loss in the discriminator based on the initial predicted image compared with the image without defogging in the image pair.

[0011] Preferably, the discriminator is obtained by weighted fusion of two sub discriminators, one of which is a multi-layer discriminator composed of an initial convolutional layer, an intermediate convolutional layer integrated with CBAM and a self-attention mechanism, and the other is a discriminator based on VGG structure composed of an intermediate convolutional layer integrated with CBAM, a self-attention mechanism and global pooling and a fully connected layer.

[0012] Preferably, the CBAM is weighted fused by channel attention and spatial attention , and the calculation formula is: ; wherein, is an input feature map, represents an element-wise multiplication.

[0013] Preferably, the alignment-invariant loss is based on a difference comparison between features output by the discriminator of the VGG structure after the same processing of the initial predicted image and the reflection-free image in the image pair as a label, respectively, based on a second VGG network.

[0014] Preferably, in the process of constructing the image pairs in the data set, the reflection region is fixed in the same sub-region, and based on the image of the sub-region, auxiliary interference terms are generated in other sub-regions of the reflection image.

[0015] Preferably, the primary feature extraction architecture is implemented using a first VGG network.

[0016] To achieve the above purpose, the application further discloses a single-image reflection removal system based on binocular vision, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the method described above when executing the computer program.

[0017] The application has the following beneficial effects:

[0018] 1. The non-aligned training mechanism is introduced into the image reflection removal task for the first time, and the overall reflection removal network architecture mainly includes a primary feature extraction architecture, an encoder-decoder architecture and a discriminator network architecture; wherein the encoder-decoder architecture is based on a series of GAFBlock for deep processing, which can be regarded as being able to dynamically capture the reflection region features through the gated attention mechanism, and after combining the non-aligned loss function, the whole reflection removal network is forced to learn the global structure consistency, rather than the simple local pixel correspondence; so that the reflection removal network can adapt to the perspective deviation and geometric deformation of the binocular image, and realize the accurate separation of the reflection layer and the background layer without explicit alignment clues, solving the adaptability problem of the traditional method in the real scene, and providing a robust reflection removal solution for complex scenes such as automatic driving.

[0019] 2. The application can accurately identify and remove irregular reflection regions in the image, whether they are located in the lower right corner of the image or any other region, and the application can effectively eliminate reflection interference, while retaining key information of the image transmission layer and effectively improving the quality of the image, laying a good foundation for subsequent feature point extraction and stereo matching.

[0020] 3. By removing the reflection region in the image through the method of the application, the number and quality of the feature points in the binocular image can be significantly improved in subsequent processing, and then the disparity estimation is more accurate, and the disparity value is closer to the true value, so as to achieve the effect of reducing the disparity error.

[0021] The application will be described in further detail below with reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification. The illustrations are shown schematically in the drawings:

[0023] Figure 1 is a schematic diagram of the principle of constructing a training data set disclosed by an embodiment of the application.

[0024] Figure 2 is a schematic diagram of the principle of dynamic masking disclosed by an embodiment of the application.

[0025] Figure 3 is a discriminator structure block diagram of a discriminant network architecture disclosed by an embodiment of the application.

[0026] Figure 4 is an anti-reflection effect comparison diagram disclosed by an embodiment of the application.

[0027] Figure 5 is a feature point extraction effect comparison disclosed by an embodiment of the application.

[0028] Figure 6 is a disparity effect improvement comparison diagram disclosed by an embodiment of the application. DETAILED DESCRIPTION

[0029] The embodiments of the application will be described in detail below with reference to the accompanying drawings, but the application can be implemented in various different ways limited and covered by the claims.

[0030] Embodiment 1

[0031] The embodiment discloses a single-image anti-reflection method based on binocular vision, including constructing a data set, dividing the data set into a training set and a validation set, constructing, training and validating an anti-reflection network structure, and deploying the trained anti-reflection network structure for testing.

[0032] In the embodiment, the data set is composed of a series of image pairs synchronously collected by binocular cameras but not aligned, and in any image pair, the non-reflection image is labeled, and the single image with reflection is labeled; preferably, in the process of constructing each image pair in the data set, the reflection region is fixed in the same sub-region, and based on the image of the sub-region, an auxiliary interference term is generated in the other sub-region of the image with reflection. The anti-reflection network structure includes a primary feature extraction architecture, an encoder-decoder architecture and a discriminant network architecture.

[0033] The primary feature extraction architecture is used for feature extraction of the input image with reflections, and then the extracted features of each layer are adjusted to the same size as the input image by bilinear interpolation, and then spliced into a super column vector with the input image with reflections, and then input into the encoder-decoder architecture. Optionally, the primary feature extraction architecture is implemented using a first VGG network.

[0034] The encoder-decoder architecture is used to perform deep processing on the up-sampling and down-sampling processes based on a series of GAFBlocks to generate an initial prediction image without reflections; wherein each GAFBlock first performs layer normalization processing on the input features, then performs channel mapping on the normalized features through a 1x1 convolution, then extracts local features through a 3x3 depth separable convolution, then adjusts the flow of feature information through a gating mechanism implemented by channel splitting and element-wise multiplication operations, and then performs SCA processing on the features processed by the gating mechanism, and then outputs the features adjusted by SCA through a 1x1 convolution. Channel integration and residual connection with input features.

[0035] The discriminator network architecture is used to discriminate the quality of the initial prediction image compared to the image without reflections in the image pair, and then generate the final prediction image without reflections by combining the discrimination result with the attention mechanism that distinguishes the reflection area and the background area.

[0036] In the training process of the de-reflection network structure, the weighted sum of the alignment-invariant loss and the generative adversarial loss of the discriminator network architecture is used as the total loss to iteratively adjust the model parameters. The alignment-invariant loss is the feature loss in the discriminator based on the initial prediction image compared to the image without reflections in the image pair.

[0037] Preferably, the discriminator is obtained by weighted fusion of two sub discriminators, one of which is a multi-layer discriminator composed of an initial convolutional layer, a CBAM-integrated intermediate convolutional layer, and a self-attention mechanism, and the other is a VGG-based discriminator composed of a CBAM-integrated intermediate convolutional layer combined with a self-attention mechanism and global pooling and a fully connected layer.

[0038] Preferably, CBAM is weighted and fused by channel attention and spatial attention The calculation formula is: ; wherein, is the input feature map, represents element-wise multiplication.

[0039] Preferably, the alignment-invariant loss is a differential comparison between the features output by the VGG-structured discriminator after processing the initial prediction image and the image without reflections in the image pair as labels based on the second VGG network.

[0040] The following further details the various parts of the embodiment:

[0041] I. Dataset production

[0042] The construction of the dynamic dataset involves extracting specific reflection regions from binocular images and introducing perturbations in different scenarios to enhance the generalization ability and adaptability of the model. The dataset is composed as shown in Figure 1 . Specifically, the reflection regions in the image can be divided into fixed reflection regions and randomly perturbed regions, where the fixed reflection regions are generated by stable reflection sources (such as vehicle glass or mirrors) and remain consistent in multiple images, while the randomly perturbed regions are affected by environmental factors and have randomness in position and shape. The image division regions are shown in Figure 2 , which shows the distribution of different reflection regions in the dataset.

[0043] In the training data generation process, first, select the fixed reflection region in each image (corresponding to the same region where the reflection region is usually fixed in the actual application scenario), and superimpose different noise patterns on it to construct the basic reflection features. Let the original image be , the fixed reflection region be , then the generation of the fixed region can be represented as: ; where ( ) is the reflection region extraction function, is a set of parameters that control the range and shape of the reflection region.

[0044] To further enhance the model's learning ability for different reflection situations and to further enhance the model's recovery ability for different regions in the image, the embodiment can introduce an auxiliary perturbation term in the image, which is obtained by randomly cropping a region of any size and position from the reflection region of the original image, used to simulate diverse image content to improve the model's generalization performance. The auxiliary perturbation term is defined as follows: ; where represents the perturbation generation function, as a perturbation parameter, including cropping position, scale change, etc.

[0045] In the training data, the fixed reflection region and the auxiliary perturbation term jointly constitute the input, and their combination form is as follows: ; where and are the weighting coefficients of the fixed region and the auxiliary perturbation term, respectively. According to experience, they can be set to , to emphasize the priority of removing the fixed region reflection, while considering the recovery of the remaining image content, in addition, N represents the total number of samples in the dataset, and D is the set of training data.

[0046] One of the objectives of the present embodiment is to train a reflection removal model based on the dataset D to recover the image after removing the reflections ; that is: n.

[0047] Based on the above data construction method, the present embodiment can effectively improve the adaptability of the model to complex reflection environments, so that it can still maintain stable reflection removal effect under different lighting conditions, viewing angles and surface material changes.

[0048] II. Network structure for removing reflections

[0049] A. The primary feature extraction architecture can be implemented using a first VGG network.

[0050] The main task of the first VGG network is to extract multi-scale deep features from the input image and fuse them with the original input data to enhance the effect of reflection removal. It can use a VGG19 network to extract features from the input image.

[0051] Define the input image I, the height and width of the image are H and W respectively, and the number of color channels of the image usually corresponds to RGB (red, green, blue) three primary colors.

[0052] Next, the VGG19 network is used to extract features from the input image I to obtain features at different levels. In this way, a series of features with different levels of abstraction can be extracted from the input image, which contains multi-level representations from low-level edge information to high-level semantic information.

[0053] In order to ensure the alignment of these features in space, the present embodiment adjusts the extracted features of each layer to the same size as the input image through bilinear interpolation, and then splices them into a super column vector with the input image with reflections to input the encoder-decoder architecture.

[0054] The process in the above section A not only enriches the expression ability of the features, but also provides multi-scale feature information for the subsequent processing stage of the network. This module enhances the network's understanding of complex scenes and provides multi-scale information for subsequent reflection removal processing.

[0055] B. Encoder-decoder architecture

[0056] This part of the architecture is used to perform deep processing based on a series of GAFBlock in the upsampling and downsampling process to generate an initial prediction image for removing reflections.

[0057] The GAFBlock module is composed of a 1x1 convolution, a 3x3 depthwise separable convolution, a simple gating mechanism, channel attention, and a residual connection. The core goal is to enhance the non-linear expression ability of features, reduce redundant calculations, and improve feature separation in the anti-reflection task.

[0058] The input features are first subjected to layer normalization, which can maintain stable normalization effects under different batch sizes, improving the adaptability and training stability of the model. Next, the features are subjected to a 1x1 convolution for channel mapping and a 3x3 depthwise separable convolution for local feature extraction, reducing computational load while preserving spatial information. The calculation formula is: ; where the output is denoted as , is the 1x1 convolution kernel, is the input feature. Next, a 3x3 depthwise separable convolution is used to further enhance spatial feature extraction capabilities, and its output is denoted as , .

[0059] Subsequently, the features enter the gating mechanism, which adjusts the flow of feature information through channel splitting and element-wise multiplication operations, allowing the network to adaptively control which features should be strengthened or suppressed. The core idea of the gating mechanism is to use information interaction between channels to enhance effective features while suppressing redundant information. In this mechanism, the input features are first split into two parts: one part is used to generate weight information, and the other part is used for feature modulation. Let the input feature be , and its channel number be C. In this embodiment, it can be evenly divided into and along the channel dimension, i.e.: ; where H and W represent the height and width of the feature map, as the information selection branch, and as the adaptive weight branch. Subsequently, the two branches are multiplied element-wise, thereby achieving feature screening and reconstruction at the channel level. The reconstruction process of the two branches can be denoted as , .

[0060] The above mechanism can be regarded as a lightweight adaptive feature screening method, where serves as a dynamic weight control information flow. Since the calculation method only involves simple channel division and element-by-element multiplication, it has lower computational overhead compared to traditional activation functions or complex attention mechanisms, and can effectively enhance the expression ability of features, especially when dealing with reflective areas, it can adaptively adjust the feature weight, so that the de-reflection network focuses more on key structural information and is less affected by irrelevant background.

[0061] features processed by the gating mechanism are input again to the simplified channel attention (SCA) module to further improve the selectivity of channel information. SCA extracts global information of the channel through global average pooling operation, and then uses 1x1 convolution for feature compression and reconstruction, thereby generating channel attention weights to weight the features The process of pooling and weighting is denoted as ; therefore: ; where represents the global average pooling operation, is a 1x1 convolution kernel. The simplified channel attention composed of pooling and weighting aims to make the network pay more attention to important channel information in the de-reflection task, and weaken the influence of background or irrelevant areas, so that the network can more accurately identify and remove the reflection area.

[0062] In addition, in order to ensure the stability of the deep network and enhance the feature expression ability of the model, the entire non-activation module adopts a residual connection structure, so that the output not only contains the features after nonlinear processing, but also retains the input information, realizing efficient fusion of features. The output of the residual connection structure is denoted as ; therefore: .

[0063] Based on the above processing, the gradient vanishing problem during training of the deep network can be effectively alleviated, while ensuring that the key structural information of the original image is not lost during the de-reflection operation. Since the entire module does not introduce an activation function, but modulates features through a gating mechanism and attention mechanism, it is more computationally efficient, avoiding the additional overhead of traditional nonlinear transformations, so that the de-reflection network can still accurately remove reflection interference and retain rich background information while maintaining efficient inference.

[0064] In the encoder-decoder architecture, the encoder is used to extract key information from the image, compress the size of the feature map, and enhance the understanding of the reflection area, while the decoder restores the size of the feature map step by step, so that the de-reflection image can effectively remove reflection interference while maintaining high-quality details. The entire network structure ensures stable gradient transmission through residual connection, and improves the selectivity of channel information by combining the simplified channel attention mechanism, thereby enhancing the de-reflection effect.

[0065] The super column vector obtained in the above A part is denoted as R, and the feature R inputs an encoder composed of multiple nonlinear activation blocks, and then gradually reduces the resolution of the feature map through a downsampling operation, while enhancing the modeling of global structural information, to obtain the encoded feature ; has: ; wherein GAFBlock() represents the operation of a nonlinear activation module, which mainly includes a gating mechanism and an attention mechanism, and Down() represents a downsampling operation, which is usually implemented by a convolutional layer to reduce the size of the feature map and increase the number of channels. Finally, the initial prediction image of the de-reflection is obtained through an upsampling and a nonlinear activation module , wherein UP() represents an upsampling operation, and the specific decoding process is: .

[0066] In the above encoder-decoder architecture, it is ensured that the background information can be preserved to the greatest extent while compressing the feature representation, and the image quality can be gradually restored in the decoding process. Through the gating mechanism and the simplified channel attention mechanism in the GAFBlock, the network can effectively enhance the recognition and suppression ability of the reflection area, so as to output a clearer initial prediction image of de-reflection.

[0067] C, discriminant network architecture.

[0068] This part is used to discriminate the quality of the initial prediction image compared with the non-reflection image in the image pair by the discriminator, and then generate the final prediction image of de-reflection by combining the attention mechanism of distinguishing the reflection area and the background area.

[0069] In order to further improve the quality of the generated image, an embodiment of the present application designs a discriminant network architecture fused with attention mechanism, i.e. image restoration network. The network includes two branches of multi-layer discriminators and discriminators based on VGG structure, as shown in Figure 3 . The multi-layer discriminators enhance the recognition ability of the reflection area by introducing the convolution block attention module (CBAM) and the self-attention mechanism, which can automatically focus on the key area and suppress the background interference. The discriminators based on VGG structure enhance the local detail capture ability through the convolution block attention module, and use the self-attention mechanism to make up for the deficiency of traditional convolution in long-distance information modeling. The overall network adopts a parallel design, and the two paths respectively extract multi-scale features and fuse the output, which significantly improves the accuracy and robustness of the discriminator, provides a more effective supervision signal for the generator, and effectively alleviates the problems of structural distortion and detail loss of the generated image.

[0070] At the same time, in order to enhance the response ability of the key area, the convolution block attention module enhances the response ability of the key area through channel attention and spatial attention The weighted fusion is performed, and the output can be represented as: ; wherein F is an input feature map, represents an element-wise multiplication, and represent channel and spatial attention mapping respectively. In addition, the self-attention mechanism is used to enhance the modeling ability of long-distance dependence, and the core calculation is: ; wherein Q, K and V represent query, key and value, is the dimension of the key vector. The whole network adopts a parallel structure design, and two paths extract multi-scale discriminative features and perform fusion, which significantly improves the accuracy and robustness of the discriminator, provides more effective supervision signals for the generator, and effectively alleviates the problems of structural distortion and detail loss.

[0071] By introducing the attention mechanism, the discriminative network shows significant advantages in processing non-aligned image defogging tasks. The attention mechanism not only improves the perception ability of the model to key areas, but also enhances the global semantic understanding and feature expression ability, significantly improving the training stability and image restoration quality. In complex background and high-resolution images, this method can more accurately distinguish the reflection area and the background area, and generate more natural and realistic defogging images, thereby making up for the limitations of traditional methods.

[0072] Among them, in the process of fusing the discriminative results to distinguish the reflection area and the background area to generate the final prediction image of the defogging by the attention mechanism, it also includes a series of processing by convolution operation to realize the change of the size of the feature map without changing the number of output channels, which is a common means for those skilled in the art and will not be described in detail.

[0073] III. Loss function

[0074] In the training process of the defogging network structure, the weighted sum of the alignment invariant loss and the generative adversarial loss of the discriminative network architecture is used as the total loss to iteratively adjust the model parameters. The alignment invariant loss is the feature loss in the discriminator based on the initial prediction image compared with the non-reflection image in the image pair.

[0075] In the loss function design, the alignment invariant loss and the generative adversarial loss are introduced. The alignment invariant loss uses high-level feature information to ensure that the network output is based on global structural information rather than local pixel points, thereby improving the defogging effect. The generative adversarial loss further improves the quality and authenticity of the generated image through the supervision of the discriminator module.

[0076] Optionally, the alignment invariant loss is: ;

[0077] wherein, represents the high-level features of the second convolutional layer of the fifth convolutional block of the second VGG network (which is independent of the first VGG network in part A, and can also use a VGG19 network), T is the target image, and is the initial prediction image generated by the encoder-decoder architecture.

[0078] The generative adversarial loss is the minimum combination of the generator loss and the discriminator loss.

[0079] In an embodiment, the weight of the alignment-invariant loss is greater than the weight of the generative adversarial loss, which helps to handle the image misalignment problem in the reflection removal task. Preferably, the weight of the former is 10 times that of the latter, which can effectively maintain the high similarity of the same pixel positions in the non-reflection area before and after removing the reflection, thereby ensuring the overall quality and consistency of the image.

[0080] In summary, the present embodiment has the following innovations:

[0081] D. For the binocular vision reflection removal task, a misalignment training mechanism is designed. The network does not rely on strict alignment of image pairs and can handle images with different viewing angles and geometric deformations. Through the misalignment loss function, the network can distinguish between reflection and background information, has good geometric adaptability and illumination robustness, and has strong generalization ability.

[0082] E. A lightweight gated attention feature block GAFBlock is proposed, which integrates depth separable convolution, gated mechanism and simplified channel attention mechanism. This module dynamically adjusts the feature weight through the gated mechanism, focuses on the reflection area, reduces computational redundancy, and improves the efficiency of reflection removal.

[0083] F. An attention-enhanced discriminator network architecture that integrates channel attention and self-attention mechanisms is designed. The discriminator network enhances its discrimination ability for the reflection area through channel attention and spatial attention, and uses the self-attention mechanism to capture long-range dependencies, improving the network's adaptability and stability to complex scenes.

[0084] G. A dynamic dataset generation method is proposed to generate reflection datasets that fit real-world scenarios, covering a variety of reflection conditions. This dataset can be used for fine-tuning and performance optimization, improving the network's adaptability and robustness in complex scenarios.

[0085] The effects mainly include:

[0086] H. The image quality is significantly improved.

[0087] In various scenarios, the dereflection network of this invention exhibits superior performance, significantly improving image quality. By accurately identifying and removing irregular reflection regions in an image, whether located in the lower right corner or anywhere else, this invention effectively eliminates reflection interference while preserving crucial information from the image transport layer. Utilizing the dereflection network of this invention effectively improves image quality, laying a solid foundation for subsequent feature point extraction and stereo matching. A comparison of the effects before and after reflection removal is provided. Figure 4 As shown, the left image is the input (the image with reflections), and the right image is the output (the effect after reflections are removed).

[0088] I. More feature points are extracted.

[0089] Feature point extraction is a crucial step in achieving tasks such as image matching, target recognition, and 3D reconstruction. This embodiment removes reflective regions from images to improve the quantity and quality of feature points extracted from binocular images. In a feature point detection comparison experiment, [the following is referenced]... Figure 5 (The left image shows a schematic diagram before reflection removal, and the right image shows a schematic diagram after reflection removal. More to come...) Figure 6 (The arrangement is consistent, so I will not elaborate further.) Experimental results show that removing the reflection area significantly improves both the quantity and quality of extracted feature points.

[0090] G. Improved stereo matching accuracy.

[0091] In stereo vision systems, the accuracy of disparity calculation is a key indicator for evaluating stereo matching performance. The method in this embodiment provides more accurate feature points for the stereo matching algorithm after de-reflection processing, resulting in more precise disparity estimation and disparity values ​​closer to the true values. During stereo matching, the algorithm employed in this invention can calculate the disparity map more accurately, reducing disparity errors.

[0092] A common semi-global matching algorithm is used for stereo image disparity calculation. Image reflection removal is performed beforehand using a reflection removal network. The disparity before and after reflection removal is compared, and the disparity comparison effect is as follows: Figure 6 As shown, the experimental results demonstrate that, compared with the unprocessed image, the disparity map after removing reflections is significantly smoother, achieving smaller disparity errors in stereo matching. This proves the rationality, reliability, and high precision of the series of processing methods in this embodiment.

[0093] Example 2

[0094] To achieve the above objectives, the present invention also discloses a single-image dereflection system based on binocular vision, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method.

[0095] In summary, the single-image reflection removal method and system based on binocular vision disclosed by the embodiments of the present application have at least the following beneficial effects:

[0096] 1. The non-aligned training mechanism is introduced into the image reflection removal task for the first time, and the overall reflection removal network architecture mainly includes a primary feature extraction architecture, an encoder-decoder architecture and a discriminator network architecture; wherein the encoder-decoder architecture is based on a series of GAFBlock for deep processing, which can be regarded as being capable of dynamically capturing reflection area features through a gated attention mechanism, and after combining a non-aligned loss function, the whole reflection removal network is forced to learn global structural consistency rather than simply local pixel correspondence; so that the reflection removal network can adapt to the perspective deviation and geometric deformation of binocular images, and realize accurate separation of the reflection layer and the background layer without explicit alignment clues, thereby solving the adaptability problem of traditional methods in real scenes and providing a robust reflection removal solution for complex scenes such as autonomous driving.

[0097] 2. The present application can accurately identify and remove irregular reflection areas in the image, whether located in the lower right corner of the image or any other area, and can effectively eliminate reflection interference while retaining key information of the image transmission layer and effectively improving the quality of the image, thereby laying a good foundation for subsequent feature point extraction and stereo matching.

[0098] 3. The reflection area in the image is removed by the method of the present application, which can significantly improve the number and quality of feature point extraction in binocular images in subsequent processing, thereby making the disparity estimation more accurate and the disparity value closer to the true value, thereby achieving the effect of reducing disparity error.

[0099] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A single-image dereflection method based on binocular vision, comprising constructing a dataset, dividing the dataset into a training set and a validation set, constructing, training, validating, and deploying the trained dereflection network structure for testing; characterized in that, The dataset consists of a series of non-aligned image pairs acquired synchronously by a series of binocular cameras. In any image pair, the non-reflective image is labeled, and the single image with reflection is marked. The dereflection network structure includes: The primary feature extraction architecture is used to extract features from the input reflective image. Then, the extracted features from each layer are adjusted to the same size as the input image through bilinear interpolation and concatenated with the input reflective image to form a hypercolumn vector, which is then input into the encoder-decoder architecture. The encoder-decoder architecture is used to perform depth processing based on a series of GAFBlocks during upsampling and downsampling to generate an initial de-reflection predicted image. Each GAFBlock first performs layer normalization on the input features. After the normalized features are processed by a 1×1 convolution for channel mapping, local features are extracted by a 3×3 depth-separable convolution. Then, a gating mechanism implemented by channel splitting and element-wise multiplication is used to adjust the flow of feature information. The gating features are then processed by SCA. After channel integration of the SCA-adjusted features by a 1×1 convolution, the residual connection is performed with the input features before output. A discriminant network architecture is used to use a discriminator to judge the quality of the initial predicted image compared to the non-reflective image in the image pair, and then combine the discrimination result with an attention mechanism that distinguishes between reflective and background regions to generate the final de-reflective predicted image; During the training of the dereflection network structure, the weighted sum of the alignment-invariant loss and the generative adversarial loss of the discriminator architecture is used as the total loss to iteratively adjust the model parameters. The alignment-invariant loss is the feature loss of the discriminator based on the initial predicted image compared to the dereflection image in the image pair.

2. The single-image dereflection method based on binocular vision according to claim 1, characterized in that, The discriminator is obtained by weighted fusion of two sub-discriminators. One sub-discriminator is a multi-layer discriminator consisting of an initial convolutional layer, an intermediate convolutional layer integrating CBAM, and a self-attention mechanism. The other sub-discriminator is a discriminator based on a VGG structure consisting of an intermediate convolutional layer integrating CBAM, a self-attention mechanism, global pooling, and a fully connected layer.

3. The single-image dereflection method based on binocular vision according to claim 2, characterized in that, The CBAM uses channel attention and spatial attention The weighted fusion is calculated using the following formula: ;in, For the input feature map, This indicates element-wise multiplication.

4. The single-image dereflection method based on binocular vision according to claim 2, characterized in that, The alignment-invariant loss is a difference comparison between the features output by the discriminator of the VGG structure after the initial predicted image and the non-reflection image used as the label in the image pair are processed in the same way based on the second VGG network.

5. The single-image dereflection method based on binocular vision according to any one of claims 1 to 4, characterized in that, In constructing each of the image pairs within the dataset, the reflective regions are fixed in the same sub-region, and auxiliary interference terms are generated in other sub-regions containing reflective images based on the image of that sub-region.

6. The single-image dereflection method based on binocular vision according to claim 5, characterized in that, The primary feature extraction architecture is implemented using a first VGG network.

7. A single-image de-reflection system based on binocular vision, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method described in any one of claims 1 to 6.

Images