Single image reflection removal method and device

Abstract

The present invention discloses a single-image reflection removal method and device, which relates to the field of image processing technology. The method comprises: using a synthetic atlas to train and verify an initial reflection removal model, and then fine-tuning the model based on a natural atlas to determine a target reflection removal model; inputting the image to be processed as a transmission flow initial image and a reflection flow initial image into a coarse separator for coarse separation, and outputting a transmission coarse prediction image and a reflection coarse prediction image respectively; using a feature encoder to determine transmission flow features and reflection flow features based on the transmission coarse prediction image and the reflection coarse prediction image respectively; using a dual-stream interactor to perform intra-stream and inter-stream feature enhancement interaction based on the transmission flow features and the reflection flow features, and correspondingly generating improved transmission flow features and improved reflection flow features; inputting the improved transmission flow features and the improved reflection flow features into an upsampling layer for upsampling, and outputting a transmission prediction image and a reflection prediction image respectively. The designed reflection removal model helps improve the reliability of image reflection removal.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a single image reflection removal method and device. Background Art

[0002] In real-world photography, image quality is often seriously affected by mirror or highlight reflections. Therefore, it is necessary to remove reflections from the image to improve image quality. In practical applications, for example, the image quality collected by smart driving cameras under complex weather conditions can be enhanced to reduce the probability of misjudgment of the smart driving system. The mirror reflection interference when shooting scenes through glass windows or other reflective surfaces can be reduced to make the captured images clearer and more realistic. The window reflection can be removed in the license plate recognition system to improve the license plate recognition accuracy.

[0003] Reflection removal is a typical ill-posed problem. The input image provides limited information, and since reflections overlap heavily with the background, the removal process can easily lead to information loss or misjudgment. Early reflection removal methods often relied on physical devices like rotating polarizers. These methods, by adjusting the angle of the polarizing filter to alter the visibility of the background scene, were often costly and ineffective in complex or dynamic scenes.

[0004] With the rapid development of deep learning technology, many neural network-based image reflection removal methods have emerged. By automatically learning the relationship between the reflection image and the target image, they can effectively extract the separation information of the reflection and transmission components in the image. For example, deep networks and perceptual losses are used to solve the problem of single image reflection removal. The ERRNet model improves the image reconstruction quality by simplifying the residual blocks and utilizing the semantic features extracted by the VGG-19 network. YTMT improves the reflection removal effect by promoting block-level communication between streams through a two-stream decomposition network. CoRRN enhances the robustness of the reflection removal model by combining image context and multi-scale information through a network structure with shared features. DSRNet adopts a two-stream cascade structure and a learnable residual module (LRM) to improve the removal effect through hierarchical semantic information and module refinement. However, existing deep learning methods still face challenges such as high computational complexity, reflection removal accuracy limited by the expressive power of the network structure, and difficulty in effectively processing high-resolution and complex reflection scenes. Further improvements are needed to improve the reliability of image reflection removal. Summary of the Invention

[0005] The present invention provides a single image reflection removal method and device, which solves the technical problem that the existing neural network-based image reflection removal method needs to further improve the reliability of image reflection removal.

[0006] A first aspect of the present invention provides a single image reflection removal method, comprising:

[0007] After training and verifying the initial reflection removal model using a synthetic atlas, the model is fine-tuned based on a natural atlas to determine a target reflection removal model; the target reflection removal model includes a coarse separator, a feature encoder, a two-stream interactor, and an upsampling layer;

[0008] Input the to-be-processed graph as the initial graph of the transmission flow and the initial graph of the reflection flow into the coarse separator for coarse separation, and output a coarse transmission prediction graph and a coarse reflection prediction graph respectively;

[0009] Performing pyramid-type multi-scale feature enhancement on the transmission coarse prediction map and the reflection coarse prediction map respectively by the feature encoder, and correspondingly determining transmission flow features and reflection flow features;

[0010] Using the dual-flow interactor to perform intra-flow and inter-flow feature enhancement interaction based on the transmission flow feature and the reflection flow feature, and correspondingly generate an improved transmission flow feature and an improved reflection flow feature;

[0011] The improved transmission flow features and the improved reflection flow features are respectively input into the upsampling layer for upsampling, and a transmission prediction map and a reflection prediction map are correspondingly output.

[0012] Optionally, the coarse separator comprises an encoder, a bottleneck layer and a decoder connected in sequence;

[0013] The encoder includes multiple cascaded residual modules and a maximum pooling layer, and the decoder includes multiple cascaded residual modules and a pixel reassembly layer. The residual modules of the encoder and decoder correspond to jump connections;

[0014] The residual module consists of multiple convolutional blocks connected in series, and the convolutional block includes a residual-connected 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function.

[0015] Optionally, the feature encoder includes a Swim Transformer module, a multi-scale feature extraction module, a cross-layer fusion module, and a 1×1 convolutional layer; the processing process of the feature encoder includes:

[0016] The feature encoding input graph is encoded layer by layer using multiple cascaded Swim Transformer modules to generate encoding features in sequence;

[0017] Each of the coding features is respectively subjected to multi-scale feature enhancement by a multi-scale feature extraction module, and corresponding enhanced features are determined;

[0018] After upsampling and fusion are performed based on the cross-layer fusion module according to each of the enhanced features, the features are input into the 1×1 convolution layer for convolution processing, and the coded stream features are output.

[0019] Optionally, the dual-stream interactor includes a bidirectional pooling feature aggregation module, a multi-scale difference feature fusion module, an intra-stream multi-scale channel interaction module, and an inter-stream information interaction module; the dual-stream interactor is used to perform intra-stream and inter-stream feature enhancement interaction based on the transmission flow feature and the reflection flow feature, and correspondingly generates an improved transmission flow feature and an improved reflection flow feature, including:

[0020] The transmission flow feature and the reflection flow feature are respectively subjected to bidirectional pooling and concentrated into global semantic information by a bidirectional pooling feature aggregation module, and a first transmission global semantic aggregation feature and a first reflection global semantic aggregation feature are correspondingly outputted;

[0021] The transmission flow feature and the first transmission global semantic aggregate feature, and the reflection flow feature and the first reflection global semantic aggregate feature are respectively input into the intra-stream multi-scale channel interaction module in pairs to perform feature interaction fusion, and generate a first transmission interaction feature and a first reflection interaction feature respectively;

[0022] Using a multi-scale combined difference feature fusion module to extract difference information from the first transmission interaction feature and the first reflection interaction feature to determine a first combined difference feature;

[0023] According to the inter-stream information interaction module, based on the proxy attention mechanism, the first combined difference feature is used as a clue to guide the first transmission interaction feature and the first reflection interaction feature to perform feature interaction, and output a first reflection contribution feature and a first transmission contribution feature;

[0024] Inputting the first transmission interaction feature and the first reflection interaction feature into a bidirectional pooling feature aggregation module respectively, and outputting a second transmission global semantic aggregation feature and a second reflection global semantic aggregation feature accordingly;

[0025] generating, by an intra-stream multi-scale channel interaction module, a second transmission interaction feature based on the first transmission interaction feature, the first reflection contribution feature, and the second transmission global semantic aggregate feature, and generating a second reflection interaction feature based on the first reflection interaction feature, the first transmission contribution feature, and the second reflection global semantic aggregate feature;

[0026] Determine a second combined difference feature according to the second transmission interaction feature and the second reflection interaction feature using a multi-scale combined difference feature fusion module;

[0027] outputting, according to the inter-stream information interaction module, a second reflection contribution feature and a second transmission contribution feature based on the second transmission interaction feature, the second reflection interaction feature, and the second combined difference feature;

[0028] Outputting a third transmission global semantic aggregation feature and a third reflection global semantic aggregation feature based on the second transmission interaction feature and the second reflection interaction feature respectively through a bidirectional pooling feature aggregation module;

[0029] Inputting the second transmission interaction feature, the second reflection contribution feature, and the third transmission global semantic aggregation feature into an intra-stream multi-scale channel interaction module to generate a third transmission interaction feature;

[0030] Inputting the second reflection interaction feature, the second transmission contribution feature, and the third reflection global semantic aggregation feature into an intra-stream multi-scale channel interaction module to determine a third reflection interaction feature;

[0031] Performing weighted fusion using the transmission flow feature, the first transmission interaction feature, the second transmission interaction feature, and the second transmission interaction feature to output an improved transmission flow feature;

[0032] A weighted fusion is performed on the reflection flow feature, the first reflection interaction feature, the second reflection interaction feature, and the third reflection interaction feature to generate an improved reflection flow feature.

[0033] Optionally, the multi-scale difference feature fusion module includes a 1×1 convolution layer, a 3×3 convolution layer, a 3×3 depth-separable convolution layer, a 5×5 depth-separable convolution layer, a 7×7 depth-separable convolution layer, a group convolution layer, and a sigmoid activation function; the processing process of the multi-scale difference feature fusion module includes:

[0034] After element-by-element subtraction of the input transmission feature and reflection feature, convolution extraction and nonlinear mapping are performed through a cascade of 3×3 convolutional layers and sigmoid activation functions to output the difference position feature;

[0035] The input transmission features are extracted in parallel using a 1×1 convolutional layer, a 3×3 depth-separable convolutional layer, a 5×5 depth-separable convolutional layer, and a 7×7 depth-separable convolutional layer. The features are then concatenated and fed into a 1×1 convolutional layer to generate transmission multi-scale features.

[0036] After multiplying the input transmission feature by the difference position feature element by element, the multi-scale transmission feature is added element by element to determine the transmission difference feature;

[0037] After extracting the input reflection features in parallel through a 1×1 convolutional layer, a 3×3 depth-separable convolutional layer, a 5×5 depth-separable convolutional layer, and a 7×7 depth-separable convolutional layer, the input is concatenated with a 1×1 convolutional layer to process the output reflection multi-scale features;

[0038] After multiplying the input reflection feature by the difference position feature element by element, the multiplication is performed element by element with the reflection multi-scale feature to determine the reflection difference feature;

[0039] The transmission difference feature and the reflection difference feature are grouped and spliced, and after inputting into the group convolution layer for feature processing, they are multiplied element-by-element with the difference position feature and processed by a 3×3 convolution layer to output the combined difference feature.

[0040] Optionally, the intra-stream multi-scale channel interaction module includes a normalization layer, a 1×1 convolution layer, a group convolution layer, a 3×3 depth-separable convolution layer, a 5×5 depth-separable convolution layer, a 7×7 depth-separable convolution layer, a softmax activation function, a GELU activation function, and a batch normalization layer; the processing process of the intra-stream multi-scale channel interaction module includes:

[0041] After the input transmission feature or reflection feature is normalized by the normalization layer, it is projected into the in-stream value key by the input 1×1 convolution layer;

[0042] After the input global semantic aggregation features are normalized by the normalization layer, they are projected into in-stream query keys and in-stream index keys respectively using a 1×1 convolutional layer, and the cosine similarity is calculated using the in-stream query keys and the in-stream index keys;

[0043] Performing attention calculation based on a softmax activation function using the in-stream query key, the in-stream index key, and the in-stream value key to determine an in-stream attention feature;

[0044] After calculating the dot product of the in-stream attention feature and the cosine similarity, the feature is added element-by-element to the input transmission feature or reflection feature to generate a self-adjusting feature;

[0045] After concatenating the input contribution features with the self-adjusting feature channels, convolution processing is performed through a group convolution layer to output in-stream fusion features;

[0046] After extracting the intra-stream fusion features in parallel through a 3×3 depthwise separable convolutional layer, a 5×5 depthwise separable convolutional layer, and a 7×7 depthwise separable convolutional layer, the features are input into a batch normalization layer for processing to determine multiple intra-stream scale features;

[0047] After concatenating the in-stream scale features with the in-stream fusion feature channels, a group convolution layer is used to perform convolution fusion to generate a deep fusion feature;

[0048] After the deep fusion feature is enhanced through a cascade of 1×1 convolutional layers, batch normalization layers, and GELU activation functions, it is added element-by-element with the deep fusion feature to output the transmission feature or reflection feature of the next layer.

[0049] Optionally, the inter-stream information interaction module includes a feature fusion block, a layer normalization layer, a two-dimensional convolution layer, a pooling layer, a proxy attention layer, a 1×1 depthwise separable convolution layer, and a gated mixing layer; the processing process of the inter-stream information interaction module includes:

[0050] The input transmission feature and the input combined difference feature, and the input reflection feature and the combined difference feature are respectively input into the feature fusion block in pairs to perform feature fusion, and generate transmission difference fusion feature and reflection difference fusion feature respectively;

[0051] Projecting the transmission difference fusion feature into a transmission index key and a transmission value key through a two-dimensional convolutional layer, and projecting the reflection difference fusion feature into a reflection index key and a reflection value key;

[0052] After the reflection difference fusion feature is normalized by a layer normalization layer, it is linearly mapped to a transmission query key through a two-dimensional convolution layer and input into a pooling layer for two-dimensional upper quarter pooling to determine a transmission proxy key;

[0053] After the transmission difference fusion feature is layer-normalized according to the layer normalization layer, it is linearly mapped to a reflection query key based on a two-dimensional convolution layer and input into a pooling layer for two-dimensional upper 1 / 4 pooling to determine a reflection proxy key;

[0054] Performing proxy attention calculation according to the transmission index key, the transmission value key, the transmission query key and the transmission proxy key through the proxy attention layer, and outputting a transmission attention feature;

[0055] Perform proxy attention calculation based on the proxy attention layer using the reflected index key, the reflected value key, the reflected query key, and the reflected proxy key, and output a reflected attention feature;

[0056] After adjusting the channel semantics of the input transmission feature using a 1×1 depthwise separable convolutional layer, the input gated mixing layer performs feature fusion with the reflection attention feature to generate the reflection contribution feature;

[0057] After adjusting the channel semantics of the input reflection feature using a 1×1 depthwise separable convolutional layer, the reflection feature is fused with the transmission attention feature through a gated mixing layer to generate the transmission contribution feature.

[0058] A second aspect of the present invention provides a single image reflection removal device, comprising:

[0059] A model training module is used to train and verify the initial reflection removal model using a synthetic atlas, and then fine-tune the model based on a natural atlas to determine a target reflection removal model; the target reflection removal model includes a coarse separator, a feature encoder, a two-stream interactor, and an upsampling layer;

[0060] An image coarse separation module is used to input the image to be processed as the initial image of the transmission flow and the initial image of the reflection flow into the coarse separator for coarse separation, and output a coarse transmission prediction image and a coarse reflection prediction image respectively;

[0061] a feature encoding module, configured to perform pyramid-type multi-scale feature enhancement on the transmission coarse prediction map and the reflection coarse prediction map respectively through the feature encoder, and correspondingly determine transmission flow features and reflection flow features;

[0062] a feature interaction module, configured to use the dual-stream interactor to perform intra-stream and inter-stream feature enhancement interaction based on the transmission flow feature and the reflection flow feature, and to generate an improved transmission flow feature and an improved reflection flow feature accordingly;

[0063] The prediction output module is used to input the improved transmission flow features and the improved reflection flow features into the upsampling layer for upsampling respectively, and output a transmission prediction map and a reflection prediction map accordingly.

[0064] A third aspect of the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the single image reflection removal method as described in any one of the above items.

[0065] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed, implements the single image reflection removal method as described in any one of the above items.

[0066] It can be seen from the above technical solutions that the present invention has the following advantages:

[0067] The above-mentioned scheme of the present invention provides a single-image reflection removal method, including: after using a synthetic atlas to train and verify the initial reflection removal model, fine-tuning the model based on the natural atlas to determine the target reflection removal model; the target reflection removal model includes a coarse separator, a feature encoder, a dual-stream interactor and an upsampling layer; the image to be processed is input into the coarse separator as the initial image of the transmission flow and the initial image of the reflection flow for coarse separation, and a transmission coarse prediction image and a reflection coarse prediction image are correspondingly output; the transmission coarse prediction image and the reflection coarse prediction image are respectively enhanced with pyramid multi-scale features by the feature encoder, and the transmission flow features and the reflection flow features are correspondingly determined; the dual-stream interactor is used to perform intra-stream and inter-stream feature enhancement interaction based on the transmission flow features and the reflection flow features, and improved transmission flow features and improved reflection flow features are correspondingly generated; the improved transmission flow features and the improved reflection flow features are respectively input into the upsampling layer for upsampling, and a transmission prediction map and a reflection prediction map are correspondingly output. Based on the above scheme, the coarse separator is used to effectively reduce the complexity of processing in the subsequent stages. The dual-stream interactor is used to perform intra-stream feature interaction and inter-stream information exchange, which effectively enhances the robustness of the model in complex reflection environments. The use of a dual-stream structure for reflection removal helps to achieve more accurate separation of reflection and transmission components. Therefore, the design-based reflection removal model helps to improve the reliability of image reflection removal. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0069] Figure 1 A flowchart of a single image reflection removal method provided by an embodiment of the present invention;

[0070] Figure 2 A schematic diagram of a process flow of a reflection removal model provided by an embodiment of the present invention;

[0071] Figure 3 A schematic structural diagram of a coarse separator provided in an embodiment of the present invention;

[0072] Figure 4 A schematic diagram of the structure of a feature encoder provided by an embodiment of the present invention;

[0073] Figure 5 Schematic diagram of the combination of a dual-stream interactor and an upsampling layer provided by an embodiment of the present invention;

[0074] Figure 6 A schematic diagram of the structure of a bidirectional pooling feature aggregation module provided by an embodiment of the present invention;

[0075] Figure 7 A schematic diagram of the structure of a multi-scale difference feature fusion module provided by an embodiment of the present invention;

[0076] Figure 8 A schematic diagram of the structure of an intra-stream multi-scale channel interaction module provided by an embodiment of the present invention;

[0077] Figure 9 A schematic diagram of the structure of an inter-flow information interaction module provided by an embodiment of the present invention;

[0078] Figure 10 This is a structural block diagram of a single image reflection removal device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0079] The embodiments of the present invention provide a single image reflection removal method and device, which are used to solve the technical problem that the reliability of image reflection removal in existing neural network-based image reflection removal methods needs to be further improved.

[0080] In order to make the purpose, features, and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0081] See also Figure 1 , Figure 1 A flowchart of the steps of a single image reflection removal method provided by an embodiment of the present invention.

[0082] This embodiment provides a single image reflection removal method, including:

[0083] Step 101: After the initial reflection removal model is trained and verified using a synthetic atlas, the model is fine-tuned based on a natural atlas to determine a target reflection removal model; the target reflection removal model includes a coarse separator, a feature encoder, a two-stream interactor, and an upsampling layer.

[0084] It should be noted that this embodiment designs a model for image reflection removal, which is called the reflection removal model. The model's architecture includes a coarse separator, a feature encoder, a two-stream interactor, and an upsampling layer.

[0085] Training the initial reflection removal model requires a large image dataset containing labeled image data, and the image dataset must be standardized and preprocessed. Standardization includes cropping, scaling, and normalization. The images are cropped while maintaining their aspect ratio, edge interference is removed, and the images are scaled to a uniform size of 224*224 (this size is set based on the model input). The pixel values of the cropped and scaled images are scaled from [0,255] to [0,1] to ensure that the image size meets the network input requirements. Data preprocessing includes basic preprocessing operations such as denoising and smoothing to provide clean input for model training.

[0086] The image dataset includes synthetic atlases and natural atlases. Each image data sample consists of a triplet consisting of a mixed map, a transmission map, and a reflection map. The difference is that the mixed map in the synthetic atlas is obtained by synthesizing any existing transmission map and reflection map through a synthesis algorithm or model. The mixed map in the natural atlas and the corresponding transmission map are derived from several public real scene datasets in the field of reflection removal, and the reflection map in the natural atlas can be obtained by subtracting the associated mixed map from the transmission map. It can be understood that the line information and texture information in the mixed map in the synthetic atlas are much more than that in the mixed map in the natural atlas, and the distribution of reflected light in the synthetic mixed map is not the same as that in the natural mixed map. In one implementation, the formula for synthesizing the mixed map can be expressed as follows: , is the first weight coefficient, is the second weight coefficient, is the virtual and real distribution range of the reflection area, is the transmission diagram, is the reflection map, is a mixed graph;

[0087] This embodiment first uses a synthetic atlas to train and verify the initial reflection removal model to determine the intermediate reflection removal model: the synthetic atlas is divided into a synthetic training set and a synthetic validation set, and the initial reflection removal model is iteratively trained with the synthetic training set as the goal of minimizing the loss function value, and the synthetic validation set is used to perform model validation and evaluation to determine the intermediate reflection removal model; thus, the weight of the round with the best effect is selected as the final model weight in the synthetic atlas stage, and the reflection removal effect and generalization ability of the model are tested using the validation set for quantitative evaluation; then, the model is fine-tuned based on the natural atlas to determine the target reflection removal model: similar to the process of the synthetic atlas, the natural atlas is divided into a natural training set and a natural validation set. , the initial reflection removal model is iteratively trained with the natural training set as the goal of minimizing the loss function value, and the natural validation set is used to perform model validation and evaluation to determine the target reflection removal model; this two-stage training method is that the natural atlas is difficult to obtain, while the synthetic atlas can be synthesized into a mixed image by any two images, and it is easier to obtain a large number of training sets containing mixed images, corresponding transmission images and corresponding reflection images. Therefore, when composing the image training set, most of the data should come from the synthetic atlas. On the other hand, the synthetic atlas is first used to train the model so that it has some texture and line separation capabilities, and then the natural atlas is used to train the model to learn the distribution of natural light in the real world, and the processing efficiency and processing effect will be better.

[0088] In a specific implementation of this embodiment, the overall loss function for calculating the loss function value combines SSIM loss, PSNR loss, and perceptual loss, which can improve the separation accuracy of reflection and transmission images while retaining the image structure and visual quality; Figure 2 It can be seen that the reflection removal model uses the mixed map as the model input and the transmission map and reflection map as the output. Therefore, the overall loss function of this embodiment includes the sum of the transmission flow total loss function and the reflection flow loss function. The transmission prediction map output by the model training and the associated real transmission map are input into the transmission flow total loss function for calculation. The reflection prediction map output by the model training and the associated real reflection map are input into the reflection flow total loss function for calculation. The calculation formula of the transmission flow total loss function or the reflection flow total loss function can be referred to: , is the total loss function, is the structural similarity loss, is the structural weight coefficient, is the peak signal-to-noise ratio loss, is the signal-to-noise weight coefficient, is the perceptual loss, is the perceptual weight coefficient. The calculation process of each loss can be specifically referred to the existing technology. Taking the transmission flow as an example, the structural similarity loss is used to measure the structural difference between the image after reflection removal and the true transmission image. The peak signal-to-noise ratio loss is used to ensure the pixel-level similarity between the removed image and the true transmission image. The perceptual loss uses a mainstream image model architecture (such as Vgg16, Vgg19, or MobileNet) to compare the semantic representations of features at different levels between the generated image and the target image.

[0089] In a specific implementation of this embodiment, in order to ensure that the reflection removal model can run efficiently on resource-constrained embedded devices, after the model training is completed, a model lightweight operation of model quantization and pruning is adopted; after estimating the size and computational complexity of the lightweight model, the trained reflection removal model is converted from the standard TensorFlow or PyTorch format to a format compatible with embedded devices through frameworks such as TensorFlow Lite and ONNX, and hardware acceleration (such as GPU, TPU, etc.) is used to optimize the model's inference process through the acceleration library of TensorFlow Lite or frameworks such as OpenVINO to further improve the running speed and efficiency. Finally, the lightweight model can be deployed on NVIDIA Jetson Nano or other devices with GPU acceleration capabilities, and images can be captured in real time through the integrated camera module and input into the target reflection removal model for processing. The processed images can be directly used in subsequent applications, such as license plate recognition, camera reflection removal, etc.

[0090] Step 102: Input the to-be-processed image as the initial image of the transmission flow and the initial image of the reflection flow into a coarse separator for coarse separation, and output a coarse transmission prediction image and a coarse reflection prediction image accordingly.

[0091] It should be noted that when the image to be processed (represented in the form of a mixed image) is received, the image to be processed is copied into two images as the initial image of the transmission flow and the initial image of the reflection flow, and then input into the coarse separator to obtain the coarsely separated transmission coarse prediction image and reflection coarse prediction image. A dual-stream architecture is initially established with the aim of alleviating the pathological nature of the input image for the subsequent network separation work.

[0092] In a specific implementation of this embodiment, the coarse separator includes an encoder, a bottleneck layer, and a decoder connected in sequence;

[0093] The encoder includes multiple cascaded residual modules and a maximum pooling layer, and the decoder includes multiple cascaded residual modules and a pixel reassembly layer. The residual modules of the encoder and decoder correspond to jump connections;

[0094] The residual module consists of multiple convolutional blocks connected in series, and the convolutional block includes a residual-connected 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function.

[0095] It should be noted that, in this embodiment, the structure of the coarse separator is as follows: Figure 3 As shown in the figure, the coarse separator uses the U-Net structure as the basic network. The encoder and decoder are connected in series through a bottleneck layer. The encoder and decoder have corresponding multi-layer residual modules. Each residual module is composed of two convolution blocks in series. Each convolution block has a "3*3Conv-BatchNorm-ReLU" structure. Residual connections are used between blocks. The processing process of each convolution block can be expressed as follows: , is the output of the convolution block, is the input of the convolution block, is the ReLU activation function, is the batch normalization layer, It is a 3×3 convolutional layer.

[0096] Step 103: Perform pyramid-type multi-scale feature enhancement on the transmission coarse prediction map and the reflection coarse prediction map respectively through a feature encoder to determine the transmission flow features and the reflection flow features accordingly.

[0097] It should be noted that the feature encoder designed in this embodiment combines the pyramid mechanism and the multi-scale feature enhancement mechanism. The transmission coarse prediction map and the reflection coarse prediction map are input into the feature encoder for feature processing and the corresponding transmission flow feature is output. and reflected flow characteristics , the transmission side and the reflection side share the parameters of an encoder, which makes it convenient for the images of both streams to be projected into the same space.

[0098] In a specific implementation of this embodiment, the feature encoder includes a Swim Transformer module, a multi-scale feature extraction module, a cross-layer fusion module, and a 1×1 convolutional layer. The processing of the feature encoder includes:

[0099] S11, using multiple cascaded Swim Transformer modules to perform feature encoding layer by layer on the feature encoding input graph, and generate encoding features in sequence;

[0100] S12, each coding feature is subjected to multi-scale feature enhancement by a multi-scale feature extraction module, and corresponding enhanced features are determined;

[0101] S13: After upsampling and fusing the enhanced features based on the cross-layer fusion module, the features are input into the 1×1 convolution layer for convolution processing, and the coded stream features are output.

[0102] It should be noted that the structure of the feature encoder of this embodiment is as follows Figure 4 As shown in the figure, the pre-trained SwimTransformer module is used as the basic network to construct a 5-layer pyramid hypercolumn feature. The Swim Transformer is used as the encoder because it can effectively capture the long-range dependency information and global semantics of the reflected image in the mixed image. At the same time, the number of parameters is less than the classic Vison Transformer. The Swim Transformer module of each layer of the feature encoder is connected to the multi-scale feature extraction module and the cross-layer fusion module. The encoded features output by the Swim Transformer model at each level are connected to the multi-scale feature extraction module, and the enhanced features from the multi-scale feature extraction module are integrated into the cross-layer fusion module, and the up-sampling features are fused from the deep layer to the shallow layer. Finally, the 1×1 convolution layer is input for convolution processing. When the transmission coarse prediction map and the reflection coarse prediction map of the dual-stream architecture are input into the feature encoder, the corresponding output encoding stream features include transmission stream features and reflection stream features.

[0103] In a more specific implementation of this embodiment, the multi-scale feature extraction module includes a 1×1 convolution layer, a 3×3 depth-separable convolution layer, a 5×5 depth-separable convolution layer, and a 7×7 depth-separable convolution layer. The processing of the multi-scale feature extraction module includes:

[0104] S1201, extract features from the input coding features using a 3×3 depthwise separable convolutional layer, a 5×5 depthwise separable convolutional layer, and a 7×7 depthwise separable convolutional layer in parallel, and then concatenate the features with the input coding features to generate scale features;

[0105] S1202: Perform convolution processing on the scale features through a 1×1 convolution layer and output corresponding enhanced features.

[0106] It should be noted that if Figure 4 As shown in the figure, the structure of the multi-scale feature extraction module is multiple parallel convolutional layers of different scales (3×3DWConv, 5×5DWConv and 7×7DWConv). The output of each scale convolutional layer is concatenated and fused through a 1×1 Conv to restore the number of channels.

[0107] In a more specific implementation of this embodiment, the cross-layer fusion module includes an upsampling layer and a depth-wise separable convolution layer; sub-step S13 includes:

[0108] S1301: Input the deepest enhanced features into the upsampling layer of the deepest cross-layer fusion module for upsampling, concatenate them with the enhanced features of the previous layer, and perform convolution processing using a depthwise separable convolution layer to generate a first fusion feature;

[0109] S1302: After the first fusion feature is input into the upsampling layer of the cross-layer fusion module of the previous layer for upsampling, it is spliced with the enhanced feature of the previous layer and convolution is performed using a depth-wise separable convolution layer. The new first fusion feature is determined and input into the cross-layer fusion module of the previous layer for upsampling, splicing and convolution processing until it is input into the 1×1 convolution layer for feature processing, and the coded stream feature is output.

[0110] It should be noted that the cross-layer fusion module is composed of "Upsample-Concat-DWConv", where the upsampling layer (Upsample) first widens the deep features by 2 times the number of channels through 1×1 convolution and then upsamples them by 2 times, so as to achieve the same size as the features of the previous layer without affecting the original number of channels. The features of the two layers are then concat- ed and fused through a depthwise separable convolution layer (DWConv) as the deep input of the cross-layer fusion module of the next layer. Finally, the fused features of the same size as the first-layer Swim Transformer module are used as the output of the feature encoder.

[0111] Step 104: A dual-flow interactor is used to perform intra-flow and inter-flow feature enhancement interaction based on the transmission flow feature and the reflection flow feature, and to generate an improved transmission flow feature and an improved reflection flow feature accordingly.

[0112] It should be noted that the encoded transmission flow features and reflected flow features will be found in the dual-stream interactor to find the features required by the convection flow in the own flow and output these features to it, thereby realizing inter-flow feature interaction. At the same time, within the own flow, the information obtained by self-feature enhancement can be combined with the information obtained by inter-flow feature interaction to perform intra-flow feature interaction within the own flow, enhance feature expression capabilities, and finally output improved transmission flow features and improved reflected flow features.

[0113] In a specific implementation of this embodiment, the dual-stream interactor includes a bidirectional pooling feature aggregation module (BPFAM), a multi-scale difference feature fusion module (MDFM), an intra-stream multi-scale channel interaction module (MSCI), and an inter-stream information interaction module (ISII). Step 104 includes the following sub-steps:

[0114] S21, through the bidirectional pooling feature aggregation module, the transmission flow features are respectively and reflected flow characteristics Perform bidirectional pooling to condense global semantic information, and output the first transmission global semantic aggregation feature. and the first reflection global semantic aggregation feature ;

[0115] S22, the transmission flow feature Aggregate features with the first transmission global semantics , reflection flow characteristics First reflection global semantic aggregation feature The multi-scale channel interaction modules in the input stream are paired to perform feature interaction fusion, and the first transmission interaction features are generated accordingly. and first reflection interaction features ;

[0116] S23, using the multi-scale combination difference feature fusion module to fusion the first transmission interaction feature and first reflection interaction features Extract difference information and determine the first combination difference feature ;

[0117] S24, based on the inter-flow information interaction module based on the agent attention mechanism to first combine the difference features First Transmission Interaction Feature for Clue Guidance Interact with First Reflection feature Perform feature interaction and output the first reflection contribution feature and the first transmission contribution characteristic ;

[0118] S25, the first transmission interaction feature and first reflection interaction features Input the bidirectional pooling feature aggregation module respectively, and output the second transmission global semantic aggregation feature and the second reflection global semantic aggregation feature ;

[0119] S26, through the multi-scale channel interaction module in the flow based on the first transmission interaction feature , first reflection contribution characteristics Aggregate features with the second transmission global semantics Generate a second transmission interaction feature , and based on the first reflection interaction feature , the first transmission contribution characteristics and the second reflection global semantic aggregation feature Generate Second Reflection Interaction Feature ;

[0120] S27, using a multi-scale combination difference feature fusion module according to the second transmission interaction feature and second reflection interaction features Determine the second combination difference feature ;

[0121] S28, according to the inter-stream information interaction module based on the second transmission interaction feature , Second reflection interaction feature and the second combination difference feature , output the second reflection contribution feature and the second transmission contribution characteristic ;

[0122] S29, through the bidirectional pooling feature aggregation module based on the second transmission interaction feature and second reflection interaction features , corresponding to the output of the third transmission global semantic aggregation feature and the third reflection global semantic aggregation feature ;

[0123] S210, set the second transmission interaction feature , Second reflection contribution characteristics Global semantic aggregation features with the third transmission Input the multi-scale channel interaction module within the stream to generate the third transmission interaction feature ;

[0124] S211, the second reflection interaction feature , the second transmission contribution characteristics and the third reflection global semantic aggregation feature Input stream multi-scale channel interaction module to determine the third reflection interaction characteristics ;

[0125] S212, using the transmission flow feature , first transmission interaction feature , the second transmission interaction feature and the second transmission interaction feature Perform weighted fusion and output improved transmission flow features ;

[0126] S213, according to the reflection flow characteristics , first reflection interaction feature , Second reflection interaction feature Interaction characteristics with the third reflection Perform weighted fusion to generate improved reflection flow features .

[0127] It should be noted that if Figure 5As shown in the figure, the encoded reflection features and transmission features will interact and fuse in the two-stream interactor, mainly through the intra-stream multi-scale channel interaction module (MSCI) and the inter-stream information interaction module (ISII). MSCI is responsible for fusing the information in the features of the other stream that is useful to the own stream with the own stream features and enhancing the features unique to the own stream. ISII is responsible for interacting the two streams, finding the features required by the other stream in the own stream in a proxy attention manner and outputting these features to the other stream. The two-stream interaction learning method based on proxy attention improves the accuracy and efficiency of image reflection removal while having low computational complexity. It can be effectively deployed on embedded devices, especially in applications such as high-resolution image processing and complex scenes. It can also run efficiently on devices with limited resources and has greater practicality.

[0128] Taking the transmission flow as the mainstream example: the characteristics of the transmission flow are output in the feature encoder. After that, when passing through MSCI for the first time, the global semantic information of the feature is condensed through the bidirectional pooling feature aggregation module (BPFAM) to obtain the first transmission global semantic aggregation feature. , and compare this vector with Input into MSCI together to get , in the same way in the reflected flow we get Then enter ISII; in ISII, first and The difference information vector of these two features is obtained by multi-scale difference feature fusion module (MDFM) , and then use this difference vector as a clue to make After full interaction, we get and , that is, the characteristic of the contribution of the reflected flow to the transmitted flow Characteristics of the contribution of the transmitted flow to the reflected flow ; In the MSCI of the transmission flow layer, collect 、 、 Process and obtain After 3 MSCI and 2 ISII, the transmission flow and reflection flow have been processed well. and The dual-stream interaction stage has three layers. In each of the first two layers, the transmission stream and the reflection stream each have an MSCI module and share an ISII module. The last layer has only an MSCI module but no ISII module. The input of each layer of the two streams is residually connected and then fused with the output of the third layer to obtain the final output. That is, in the transmission stream, 、 、 、 Improved transmission flow characteristics are obtained after weighted fusion , expressed in formula: , which is expressed as improved reflection flow characteristics in the reflection flow , here , is the feature indexing layer in the two-stream interactor, is the fusion weight, is the transmission characteristic of the transmission flow in the two-flow interactor, is the reflection characteristic of the reflected flow in the two-flow interactor.

[0129] In a more specific implementation of this embodiment, the bidirectional pooling feature aggregation module (BPFAM) includes a pooling layer, a normalization layer, a 1×1 convolution layer, a GELU activation function, and a gating layer. The processing of the bidirectional pooling feature aggregation module includes:

[0130] S31, pooling the input transmission features or reflection features along the width direction and length direction respectively through the pooling layer to generate width pooling features and length pooling features;

[0131] S32, using a cascaded normalization layer, a 1×1 convolution layer, and a GELU activation function to perform normalization, convolution extraction, and nonlinear mapping on the width pooling features in turn to determine the width semantic features;

[0132] S33, after inputting the width pooling feature into the 1×1 convolution layer for feature processing, it is added element by element with the width semantic feature to output the width semantic aggregation feature;

[0133] S34, using cascaded normalization layers, 1×1 convolution layers, and GELU activation functions to normalize, convolute, and nonlinearly map the length pooling features to generate length semantic features;

[0134] S35, input the length pooling feature into the 1×1 convolution layer for feature processing, and then add it to the length semantic feature to determine the length semantic aggregation feature;

[0135] S36. Adaptively fuse the width semantic aggregation features and the length semantic aggregation features through the gating layer to output the global semantic aggregation features.

[0136] It should be noted that the input of the MSCI module requires the use of the BPFAM module, such as Figure 6 As shown, the BPFAM module inherits the transmission characteristics of the transmission flow ( 、 and ) or the reflection characteristics of the reflection flow ( 、 and ), and then perform pooling along the length direction (h-pooling) and the width direction (w-pooling) respectively. The features pooled along different directions are respectively passed through the residual connection feature extraction block for channel feature semantic adjustment. The feature extraction block includes a cascaded normalization layer with a 1×1 convolution layer residual connection, a 1×1 convolution layer and a GELU activation function, so as to obtain the semantic aggregation features in this direction. Finally, the semantic aggregation features of the two directions are adaptively fused through a gating mechanism to obtain the channel-level global semantic aggregation features.

[0137] It is understood that the transmission characteristics in this embodiment can refer to the characteristics of the transmission flow in the dual-flow interactor. 、 、 or , the reflection feature can refer to the feature of the reflection flow in the two-stream interactor 、 、 or .

[0138] In a more specific implementation of this embodiment, the multi-scale difference feature fusion module (MDFM) includes a 1×1 convolution layer, a 3×3 convolution layer, a 3×3 depth-separable convolution layer, a 5×5 depth-separable convolution layer, a 7×7 depth-separable convolution layer, a group convolution layer, and a sigmoid activation function. The processing of the multi-scale difference feature fusion module includes:

[0139] S41, after element-by-element subtraction of the input transmission feature and reflection feature, convolution extraction and nonlinear mapping are performed through a cascaded 3×3 convolution layer and a sigmoid activation function to output the difference position feature;

[0140] S42, use 1×1 convolution layer, 3×3 depth-separable convolution layer, 5×5 depth-separable convolution layer and 7×7 depth-separable convolution layer to extract the input transmission features in parallel, then splice them and input them into 1×1 convolution layer to generate transmission multi-scale features;

[0141] S43, multiplying the input transmission feature and the difference position feature element by element, and then adding the result to the transmission multi-scale feature element by element to determine the transmission difference feature;

[0142] S44, after extracting the input reflection features in parallel through a 1×1 convolution layer, a 3×3 depth-separable convolution layer, a 5×5 depth-separable convolution layer, and a 7×7 depth-separable convolution layer, the input is spliced into a 1×1 convolution layer to process the output reflection multi-scale features;

[0143] S45, multiplying the input reflection feature and the difference position feature element by element, and then adding the result to the reflection multi-scale feature element by element to determine the reflection difference feature;

[0144] S46. Group the transmission difference features and the reflection difference features and concatenate them, input them into the group convolution layer for feature processing, multiply them element-by-element with the difference position features, and use a 3×3 convolution layer to process and output the combined difference features.

[0145] It should be noted that the ISII module needs to use the MDFM module to find the difference information between the two streams as a clue to guide the information interaction between the streams; Figure 7 As shown, in MDFM, the transmission characteristics of the transmission flow input are first (Here refers to or ) and the reflection characteristics of the reflected flow input (Here refers to or ) by pixel-by-pixel subtraction, Conv and sigmoid operations initially extract the spatial distribution of the difference position on the image. This process is expressed as , is the feature indexing layer in the two-stream interactor, is the difference position feature, is a 3×3 convolutional layer, is the sigmoid activation function; then the features of the two streams are fused with multi-scale features through convolution kernels of different scales (1×1Conv, 3×3DWConv, 5×5DWConv and 7×7DWConv), so that the subsequent fusion of difference information can pay more attention to the impact of spatial differences. For example, the process can be expressed as the transmission multi-scale feature , is a 1×1 convolutional layer, is a 3×3 depthwise separable convolutional layer, is a 5×5 depth-wise separable convolutional layer, is a 7×7 depthwise separable convolutional layer, The first step is to splice the difference position features extracted in the first step with the features of the original input and then add them to the vector extracted by multi-scale features. This can enhance the intensity of the difference position feature pixels on the basis of the original input and highlight the difference information. Finally, the features that highlight the difference information of the two streams are group-spliced and fused using group convolution. After multiplying them with the difference position features, they are processed through a 3×3 convolution layer to output the combined difference features, thereby obtaining the common difference features of the two streams.

[0146] In a more specific implementation of this embodiment, the intra-stream multi-scale channel interaction module (MSCI) includes a normalization layer, a 1×1 convolution layer, a group convolution layer, a 3×3 depth-separable convolution layer, a 5×5 depth-separable convolution layer, a 7×7 depth-separable convolution layer, a softmax activation function, a GELU activation function, and a batch normalization layer; the processing process of the intra-stream multi-scale channel interaction module includes:

[0147] S51, after normalizing the input transmission feature or reflection feature through the normalization layer, input the 1×1 convolution layer to project it into the in-stream value key;

[0148] S52, after normalizing the input global semantic aggregation features through a normalization layer, a 1×1 convolutional layer is used to project them into an intra-stream query key and an intra-stream index key, and the cosine similarity is calculated using the intra-stream query key and the intra-stream index key;

[0149] S53, performing attention calculation based on the softmax activation function using the in-stream query key, the in-stream index key, and the in-stream value key to determine the in-stream attention feature;

[0150] S54, after calculating the dot product of the in-stream attention feature and the cosine similarity, the feature is added element by element to the input transmission feature or reflection feature to generate a self-adjusting feature;

[0151] S55, after concatenating the input contribution features with the self-adjusting feature channels, perform convolution processing through the group convolution layer and output the in-stream fusion features;

[0152] S56, after extracting parallel features of the intra-stream fusion features through a 3×3 depth-separable convolutional layer, a 5×5 depth-separable convolutional layer, and a 7×7 depth-separable convolutional layer, the features are input into a batch normalization layer to determine multiple intra-stream scale features;

[0153] S57, after concatenating the scale features within each flow with the fusion feature channels within the flow, a group convolution layer is used to perform convolution fusion to generate a deep fusion feature;

[0154] S58. After the deep fusion feature is enhanced through the cascaded 1×1 convolution layer, batch normalization layer and GELU activation function, it is added element by element with the deep fusion feature to output the transmission feature or reflection feature of the next layer.

[0155] It should be noted that if Figure 8 As shown in the figure, the multi-scale channel interaction module in the flow generally accepts three inputs. Taking the transmission flow as an example, the input includes the transmission feature , global semantic aggregation features and contribution characteristics ,The intra-stream multi-scale channel interaction module can extract image features from multiple scales to improve the model’s sensitivity and robustness to information at different scales;

[0156] The obtained global semantic aggregation feature condenses the information in the space into one value, but the information between channels is not fully communicated. Therefore, this global semantic aggregation feature is projected as the query key Q and the index key K. The projection is a value key V so that attention calculation can be performed. A weighted residual connection is also used. That is, the attention adjustment feature is weighted according to the overall cosine similarity of the query key Q and the index key K. The purpose is to adjust the information between channels according to the similarity. The output of this process is the self-adjusted feature of the transmission flow , is the cosine inner product, It is the square root of the K dimension for temperature control. is the softmax activation function, is the transpose of the index key;

[0157] However, Just adjust itself to its own information, so it can incorporate information from another stream To better recover the transmission map from the mixed map; first and Perform channel-level concat and grouping Convolution is used to initially and briefly fuse the two features. Next, multi-scale feature extraction is performed through deep convolution kernels of different scales (3×3DWConv, 5×5DWConv and 7×7DWConv), and then channel-level skip concat and grouping are performed. Convolution is used to fuse features of different scales, and finally a residual connection feature enhancement block (1×1Conv-BatchNorm-GELU) is used to condense the features just extracted. The processing of the feature enhancement block can be expressed as the flow feature of the next layer of the transmission flow. , is the feature indexing layer in the two-stream interactor, For deep fusion features, is the GELU activation function, is a 1×1 convolutional layer, is the batch normalization layer.

[0158] It is understandable that Figure 8 The "R" in the function refers to the reshape operation, which changes the feature shape to meet the shape required by matrix multiplication, and is set according to the corresponding feature processing.

[0159] In a more specific implementation of this embodiment, the inter-stream information interaction module (ISII) includes a feature fusion block, a layer normalization layer, a two-dimensional convolution layer, a pooling layer, a proxy attention layer, a 1×1 depthwise separable convolution layer, and a gated mixing layer. The processing of the inter-stream information interaction module includes:

[0160] The input transmission feature and the input combination difference feature, the input reflection feature and the combination difference feature are respectively input into the feature fusion block for feature fusion, and the corresponding transmission difference fusion feature is generated. and reflection difference fusion features ;

[0161] The transmission difference fusion features are obtained through a two-dimensional convolutional layer. Projection into transmission index key and transmission value key, and fusion feature of reflection difference Projection is reflected index key and reflected value key;

[0162] Adopt layer normalization layer to fuse reflection difference features After layer normalization, the two-dimensional convolutional layer is linearly mapped to the transmission query key and input into the pooling layer for two-dimensional upper 1 / 4 pooling to determine the transmission proxy key;

[0163] Fusion features based on layer-normalized layer-pair transmission differences After layer normalization, the two-dimensional convolutional layer is linearly mapped to the reflection query key and input into the pooling layer for two-dimensional upper quarter pooling to determine the reflection proxy key;

[0164] The proxy attention layer calculates the proxy attention based on the transmission index key, transmission value key, transmission query key and transmission proxy key, and outputs the transmission attention feature;

[0165] Based on the proxy attention layer, the reflected index key, reflected value key, reflected query key and reflected proxy key are used to calculate the proxy attention and output the reflected attention feature;

[0166] After adjusting the channel semantics of the input transmission features using a 1×1 depthwise separable convolutional layer, the input gated mixing layer fuses the reflection attention features to generate reflection contribution features.

[0167] After adjusting the channel semantics of the input reflection feature using a 1×1 depthwise separable convolutional layer, the transmission contribution feature is generated by fusing it with the transmission attention feature through a gated mixing layer.

[0168] It should be noted that at this stage, Figure 9 As shown, the transmission characteristics of the transmission flow With reflection flow In combination with different features Under the guidance of , the agent attention mechanism is used to interact. Taking the transmission flow to the reflection flow as an example: first and Obtained by fusion through feature fusion block (fusion) ), and in the same way we get After concat two features, the number of channels is the sum of the number of channels of the two features. The 1×1 convolution reduces the number of channels and allows information exchange between channels. After linear transformation, we get the index key K and value key V, and then After layer normalization and linear mapping operations as Q key, and after two-dimensional 1 / 4 pooling of Q key features, the proxy key Agent is obtained, which is the core of the proxy attention. The proxy attention is different from the ordinary self-attention mechanism. The ordinary self-attention mechanism can be expressed as , is the attention mechanism, is the attention output, and the proxy attention consists of two Softmax attention operations, namely proxy aggregation and proxy broadcast, which can be expressed as , that is, first consider the proxy tag A as the query key and calculate the attention between A, K and V to aggregate the proxy features from all values V , then treat A as the index key, As the value key, together with the query key Q in the second attention calculation, the global information from the agent feature is broadcast to each query token, and finally the output is obtained The entire agent attention process will add agent bias (Agent Bias) or the first agent bias and second-agent bias , so the complete formula is: , is the transpose of the surrogate key, where That is Towards Query what information is useful to you, and finally compare it with the original input After the output residual connection of 1×1DWConv to adjust the channel semantics, After the gating mechanism is fused into , , TR refers to the T to R characteristic, which represents the information transmitted from the transmitted flow to the reflected flow. The gate mixing operation can be expressed as: ; This operation is symmetrical and we get , representing the information transmitted from the reflected flow to the transmitted flow.

[0169] It is understandable that Figure 9 The "P" in the formula refers to four-to-one pooling, and the general feature shape is square * number of channels. Quarter pooling can select representative pixels with relatively small error. If you want to maintain the square feature, in addition to quarter pooling, usually only one-sixteenth pooling is left, which is easy to increase the error. Pooling is to downsample feature Q into feature A, so that the computational complexity is greatly reduced when the attention mechanism is performed later. For example, if Q is (256, 256, C), K is (256, 256, C), and A is (128, 128, C) after pooling by Q, the computational complexity A*K+A*Q of this embodiment is 128*128*256*256*2, and the computational complexity Q*K of the classic attention mechanism is 256*256*256*256. It can be seen that the method of this embodiment has halved the computational complexity.

[0170] Step 105: Input the improved transmission flow features and the improved reflection flow features into the upsampling layer for upsampling, and output a transmission prediction map and a reflection prediction map respectively.

[0171] For example, the above method embodiment can be deployed in the following scenarios:

[0172] Scenario 1: At night, the intelligent driving system camera may obtain images reflected from the front windshield under the influence of the light source inside the car, affecting the judgment of the intelligent driving system. In the vehicle-mounted system, the above-mentioned target reflection removal model is implanted in the camera and other embedded imaging devices to execute steps 102-105, thereby processing the window reflection in real time to ensure that the judgment of the intelligent driving system is not affected by the reflected image.

[0173] Scenario 2: When the visual obstacle avoidance function equipped with a drone is used in strong daylight, especially when photographing lakes, water surfaces, or areas with glass windows, mirror reflection problems often occur. The reflections in these scenarios will affect the visual obstacle avoidance function's ability to accurately judge the distance to obstacles, causing unnecessary emergency braking and affecting user experience. By embedding the above-mentioned target reflection removal model in the drone's camera and other embedded imaging devices to execute steps 102-105, the captured images can be processed in real time and reflections can be removed, providing clear image data to ensure the normal operation of the visual obstacle avoidance function.

[0174] In an embodiment of the present invention, a coarse separator is used to effectively reduce the complexity of processing required in subsequent stages, significantly reduce the computational burden, accelerate the training and inference processes, and a dual-stream interactor is used to perform intra-stream feature interaction and inter-stream information exchange, effectively enhancing the robustness of the model in complex reflection environments. The use of a dual-stream structure for reflection removal helps to achieve more accurate separation of reflection and transmission components. Therefore, the overall design-based reflection removal model helps to improve the reliability of image reflection removal.

[0175] See also Figure 10 , Figure 10 This is a structural block diagram of a single image reflection removal device provided by an embodiment of the present invention.

[0176] The present invention provides a single image reflection removal device, comprising:

[0177] The model training module 1001 is used to train and verify the initial reflection removal model using a synthetic atlas, and then fine-tune the model based on a natural atlas to determine a target reflection removal model; the target reflection removal model includes a coarse separator, a feature encoder, a two-stream interactor, and an upsampling layer;

[0178] The image coarse separation module 1002 is used to input the image to be processed as the initial image of the transmission flow and the initial image of the reflection flow into the coarse separator for coarse separation, and output a coarse transmission prediction image and a coarse reflection prediction image respectively;

[0179] A feature encoding module 1003 is configured to perform pyramid-type multi-scale feature enhancement on the transmission coarse prediction map and the reflection coarse prediction map respectively through a feature encoder, and determine transmission flow features and reflection flow features accordingly;

[0180] A feature interaction module 1004 is configured to use a dual-stream interactor to perform intra-stream and inter-stream feature enhancement interaction based on the transmission flow feature and the reflection flow feature, and to generate an improved transmission flow feature and an improved reflection flow feature accordingly;

[0181] The prediction output module 1005 is used to input the improved transmission flow features and the improved reflection flow features into the upsampling layer for upsampling, and output a transmission prediction map and a reflection prediction map respectively.

[0182] Furthermore, the coarse separator includes an encoder, a bottleneck layer, and a decoder connected in sequence;

[0183] The encoder includes multiple cascaded residual modules and a maximum pooling layer, and the decoder includes multiple cascaded residual modules and a pixel reassembly layer. The residual modules of the encoder and decoder correspond to jump connections;

[0184] The residual module consists of multiple convolutional blocks connected in series, and the convolutional block includes a residual-connected 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function.

[0185] Furthermore, the feature encoder includes a Swim Transformer module, a multi-scale feature extraction module, a cross-layer fusion module, and a 1×1 convolutional layer; the processing of the feature encoder includes:

[0186] The feature encoding input graph is encoded layer by layer using multiple cascaded Swim Transformer modules to generate encoding features in sequence;

[0187] Each coding feature is enhanced by a multi-scale feature extraction module, and the enhanced features are determined accordingly;

[0188] After upsampling and fusion based on the cross-layer fusion module according to each enhanced feature, the 1×1 convolution layer is input for convolution processing to output the coded stream features.

[0189] Furthermore, the dual-stream interactor includes a bidirectional pooling feature aggregation module, a multi-scale difference feature fusion module, an intra-stream multi-scale channel interaction module, and an inter-stream information interaction module; the feature interaction module 1004 is specifically used to:

[0190] The bidirectional pooling feature aggregation module performs bidirectional pooling on the transmission flow features and the reflection flow features to condense the global semantic information, and outputs the first transmission global semantic aggregation feature and the first reflection global semantic aggregation feature respectively;

[0191] The transmission flow features and the first transmission global semantic aggregation features, and the reflection flow features and the first reflection global semantic aggregation features are respectively input into the multi-scale channel interaction module within the stream for feature interaction fusion, and the first transmission interaction features and the first reflection interaction features are generated respectively;

[0192] A multi-scale combined difference feature fusion module is used to extract difference information from the first transmission interaction feature and the first reflection interaction feature to determine a first combined difference feature;

[0193] According to the inter-stream information interaction module, based on the proxy attention mechanism, the first combination difference feature is used as a clue to guide the first transmission interaction feature and the first reflection interaction feature to interact with each other, and output the first reflection contribution feature and the first transmission contribution feature;

[0194] The first transmission interaction feature and the first reflection interaction feature are respectively input into the bidirectional pooling feature aggregation module, and the second transmission global semantic aggregation feature and the second reflection global semantic aggregation feature are correspondingly output;

[0195] Generate a second transmission interaction feature based on the first transmission interaction feature, the first reflection contribution feature, and the second transmission global semantic aggregation feature through the intra-stream multi-scale channel interaction module, and generate a second reflection interaction feature based on the first reflection interaction feature, the first transmission contribution feature, and the second reflection global semantic aggregation feature;

[0196] Determine a second combined difference feature based on the second transmission interaction feature and the second reflection interaction feature using a multi-scale combined difference feature fusion module;

[0197] Outputting a second reflection contribution feature and a second transmission contribution feature according to the inter-stream information interaction module based on the second transmission interaction feature, the second reflection interaction feature and the second combined difference feature;

[0198] The bidirectional pooling feature aggregation module outputs a third transmission global semantic aggregation feature and a third reflection global semantic aggregation feature based on the second transmission interaction feature and the second reflection interaction feature respectively;

[0199] Inputting the second transmission interaction feature, the second reflection contribution feature, and the third transmission global semantic aggregation feature into the intra-stream multi-scale channel interaction module to generate a third transmission interaction feature;

[0200] Inputting the second reflection interaction feature, the second transmission contribution feature, and the third reflection global semantic aggregation feature into the intra-stream multi-scale channel interaction module to determine the third reflection interaction feature;

[0201] The transmission flow feature, the first transmission interaction feature, the second transmission interaction feature and the second transmission interaction feature are weightedly fused to output an improved transmission flow feature;

[0202] The improved reflection flow feature is generated by performing weighted fusion on the reflection flow feature, the first reflection interaction feature, the second reflection interaction feature and the third reflection interaction feature.

[0203] Furthermore, the multi-scale difference feature fusion module includes a 1×1 convolution layer, a 3×3 convolution layer, a 3×3 depth-separable convolution layer, a 5×5 depth-separable convolution layer, a 7×7 depth-separable convolution layer, a group convolution layer, and a sigmoid activation function. The processing process of the multi-scale difference feature fusion module includes:

[0204] After element-by-element subtraction of the input transmission feature and reflection feature, convolution extraction and nonlinear mapping are performed through a cascade of 3×3 convolutional layers and sigmoid activation functions to output the difference position feature;

[0205] The input transmission features are extracted in parallel using a 1×1 convolutional layer, a 3×3 depth-separable convolutional layer, a 5×5 depth-separable convolutional layer, and a 7×7 depth-separable convolutional layer. The features are then concatenated and fed into a 1×1 convolutional layer to generate transmission multi-scale features.

[0206] After multiplying the input transmission feature and the difference position feature element by element, the result is added to the transmission multi-scale feature element by element to determine the transmission difference feature;

[0207] After extracting the input reflection features in parallel through a 1×1 convolutional layer, a 3×3 depth-separable convolutional layer, a 5×5 depth-separable convolutional layer, and a 7×7 depth-separable convolutional layer, the input is concatenated with a 1×1 convolutional layer to process the output reflection multi-scale features;

[0208] The input reflection feature is multiplied by the difference position feature element by element, and then added to the reflection multi-scale feature element by element to determine the reflection difference feature;

[0209] The transmission difference features and the reflection difference features are grouped and spliced, and after inputting into the group convolution layer for feature processing, they are multiplied element-by-element with the difference position features and processed by a 3×3 convolution layer to output the combined difference features.

[0210] Furthermore, the intra-stream multi-scale channel interaction module includes a normalization layer, a 1×1 convolution layer, a group convolution layer, a 3×3 depthwise separable convolution layer, a 5×5 depthwise separable convolution layer, a 7×7 depthwise separable convolution layer, a softmax activation function, a GELU activation function, and a batch normalization layer. The processing process of the intra-stream multi-scale channel interaction module includes:

[0211] After the input transmission feature or reflection feature is normalized by the normalization layer, it is projected into the in-stream value key by the input 1×1 convolution layer;

[0212] After the input global semantic aggregation features are normalized by the normalization layer, they are projected into in-stream query keys and in-stream index keys respectively using a 1×1 convolutional layer, and the cosine similarity is calculated using the in-stream query keys and in-stream index keys.

[0213] Based on the softmax activation function, the in-stream query key, in-stream index key and in-stream value key are used to calculate the attention and determine the in-stream attention feature;

[0214] After calculating the dot product of the in-stream attention feature and the cosine similarity, it is added element-by-element to the input transmission feature or reflection feature to generate a self-adjusting feature;

[0215] After concatenating the input contribution features with the self-adjusting feature channels, the convolution layer performs convolution processing and outputs the in-stream fusion features.

[0216] After parallel feature extraction of intra-stream fusion features through 3×3 depth-wise separable convolutional layer, 5×5 depth-wise separable convolutional layer and 7×7 depth-wise separable convolutional layer, the features are input into batch normalization layer to determine multiple intra-stream scale features;

[0217] After concatenating the scale features within each flow with the fusion feature channels within the flow, a group convolution layer is used for convolution fusion to generate deep fusion features.

[0218] After the deep fusion feature is enhanced through the cascaded 1×1 convolution layer, batch normalization layer and GELU activation function, it is added element by element with the deep fusion feature to output the transmission feature or reflection feature of the next layer.

[0219] Furthermore, the inter-stream information interaction module includes a feature fusion block, a layer normalization layer, a two-dimensional convolution layer, a pooling layer, a proxy attention layer, a 1×1 depthwise separable convolution layer, and a gated mixing layer. The processing of the inter-stream information interaction module includes:

[0220] The input transmission feature and the input combination difference feature, as well as the input reflection feature and the combination difference feature are respectively input into the feature fusion block in pairs to perform feature fusion, and generate transmission difference fusion feature and reflection difference fusion feature respectively;

[0221] Project the transmission difference fusion features into transmission index keys and transmission value keys through a two-dimensional convolutional layer, and project the reflection difference fusion features into reflection index keys and reflection value keys;

[0222] After the reflection difference fusion features are normalized by the layer normalization layer, they are linearly mapped to the transmission query key through the two-dimensional convolution layer and input into the pooling layer for two-dimensional upper 1 / 4 pooling to determine the transmission proxy key;

[0223] After the transmission difference fusion features are layer-normalized by the layer normalization layer, they are linearly mapped to the reflection query key based on the two-dimensional convolution layer and input into the pooling layer for two-dimensional upper 1 / 4 pooling to determine the reflection proxy key;

[0224] The proxy attention layer calculates the proxy attention based on the transmission index key, transmission value key, transmission query key and transmission proxy key, and outputs the transmission attention feature;

[0225] Based on the proxy attention layer, the reflected index key, reflected value key, reflected query key and reflected proxy key are used to calculate the proxy attention and output the reflected attention feature;

[0226] After adjusting the channel semantics of the input transmission features using a 1×1 depthwise separable convolutional layer, the input gated mixing layer fuses the reflection attention features to generate reflection contribution features.

[0227] After adjusting the channel semantics of the input reflection feature using a 1×1 depthwise separable convolutional layer, the transmission contribution feature is generated by fusing it with the transmission attention feature through a gated mixing layer.

[0228] An embodiment of the present invention further provides a computer device comprising a memory and a processor, wherein a computer program is stored in the memory; when the computer program is executed by the processor, the processor executes the steps of the single image reflection removal method of any of the above embodiments.

[0229] An embodiment of the present invention further provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the steps of the single image reflection removal method as described in any of the above embodiments are implemented.

[0230] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0231] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0232] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0233] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0234] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0235] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A single image reflection removal method, characterized in that: include: After the initial reflection removal model is trained and verified using a synthetic atlas, the model is fine-tuned based on a natural atlas to determine the target reflection removal model. The target reflection removal model includes a coarse separator, a feature encoder, a two-stream interactor and an upsampling layer; Input the to-be-processed graph as the initial graph of the transmission flow and the initial graph of the reflection flow into the coarse separator for coarse separation, and output a coarse transmission prediction graph and a coarse reflection prediction graph respectively; Performing pyramid-type multi-scale feature enhancement on the transmission coarse prediction map and the reflection coarse prediction map respectively by the feature encoder, and correspondingly determining transmission flow features and reflection flow features; Using the dual-flow interactor to perform intra-flow and inter-flow feature enhancement interaction based on the transmission flow feature and the reflection flow feature, and correspondingly generate an improved transmission flow feature and an improved reflection flow feature; Inputting the improved transmission flow feature and the improved reflection flow feature into an upsampling layer for upsampling respectively, and outputting a transmission prediction map and a reflection prediction map correspondingly; The dual-stream interactor includes a bidirectional pooling feature aggregation module, a multi-scale difference feature fusion module, an intra-stream multi-scale channel interaction module, and an inter-stream information interaction module; The bidirectional pooling feature aggregation module includes a pooling layer, a normalization layer, a 1×1 convolution layer, a GELU activation function and a gating layer; The multi-scale difference feature fusion module includes a 1×1 convolution layer, a 3×3 convolution layer, a 3×3 depth-separable convolution layer, a 5×5 depth-separable convolution layer, a 7×7 depth-separable convolution layer, a group convolution layer and a sigmoid activation function; The intra-stream multi-scale channel interaction module includes a normalization layer, a 1×1 convolution layer, a group convolution layer, a 3×3 depth-separable convolution layer, a 5×5 depth-separable convolution layer, a 7×7 depth-separable convolution layer, a softmax activation function, a GELU activation function, and a batch normalization layer; The inter-stream information interaction module includes a feature fusion block, a layer normalization layer, a two-dimensional convolution layer, a pooling layer, a proxy attention layer, a 1×1 depthwise separable convolution layer, and a gated mixing layer.

2. The single image reflection removal method according to claim 1, characterized in that: The coarse separator includes an encoder, a bottleneck layer and a decoder connected in sequence; The encoder includes multiple cascaded residual modules and a maximum pooling layer, and the decoder includes multiple cascaded residual modules and a pixel reassembly layer. The residual modules of the encoder and decoder correspond to jump connections; The residual module consists of multiple convolutional blocks connected in series, and the convolutional block includes a residual-connected 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function.

3. The single image reflection removal method according to claim 1, characterized in that: The feature encoder includes a SwimTransformer module, a multi-scale feature extraction module, a cross-layer fusion module and a 1×1 convolutional layer; the processing process of the feature encoder includes: The feature encoding input graph is encoded layer by layer using multiple cascaded Swim Transformer modules to generate encoding features in sequence; Each of the coding features is respectively subjected to multi-scale feature enhancement by a multi-scale feature extraction module, and corresponding enhanced features are determined; After upsampling and fusion are performed based on the cross-layer fusion module according to each of the enhanced features, the features are input into the 1×1 convolution layer for convolution processing, and the coded stream features are output.

4. The single image reflection removal method according to claim 1, characterized in that: The dual-flow interactor is used to perform intra-flow and inter-flow feature enhancement interaction according to the transmission flow feature and the reflection flow feature, and correspondingly generates an improved transmission flow feature and an improved reflection flow feature, including: The transmission flow feature and the reflection flow feature are respectively subjected to bidirectional pooling and concentrated into global semantic information by a bidirectional pooling feature aggregation module, and a first transmission global semantic aggregation feature and a first reflection global semantic aggregation feature are correspondingly outputted; The transmission flow feature and the first transmission global semantic aggregate feature, and the reflection flow feature and the first reflection global semantic aggregate feature are respectively input into the intra-stream multi-scale channel interaction module in pairs to perform feature interaction fusion, and generate a first transmission interaction feature and a first reflection interaction feature respectively; Using a multi-scale combined difference feature fusion module to extract difference information from the first transmission interaction feature and the first reflection interaction feature to determine a first combined difference feature; According to the inter-stream information interaction module, based on the proxy attention mechanism, the first combined difference feature is used as a clue to guide the first transmission interaction feature and the first reflection interaction feature to perform feature interaction, and output a first reflection contribution feature and a first transmission contribution feature; Inputting the first transmission interaction feature and the first reflection interaction feature into a bidirectional pooling feature aggregation module respectively, and outputting a second transmission global semantic aggregation feature and a second reflection global semantic aggregation feature accordingly; generating, by an intra-stream multi-scale channel interaction module, a second transmission interaction feature based on the first transmission interaction feature, the first reflection contribution feature, and the second transmission global semantic aggregate feature, and generating a second reflection interaction feature based on the first reflection interaction feature, the first transmission contribution feature, and the second reflection global semantic aggregate feature; Determine a second combined difference feature according to the second transmission interaction feature and the second reflection interaction feature using a multi-scale combined difference feature fusion module; outputting, according to the inter-stream information interaction module, a second reflection contribution feature and a second transmission contribution feature based on the second transmission interaction feature, the second reflection interaction feature, and the second combined difference feature; Outputting a third transmission global semantic aggregation feature and a third reflection global semantic aggregation feature based on the second transmission interaction feature and the second reflection interaction feature respectively through a bidirectional pooling feature aggregation module; Inputting the second transmission interaction feature, the second reflection contribution feature, and the third transmission global semantic aggregation feature into an intra-stream multi-scale channel interaction module to generate a third transmission interaction feature; Inputting the second reflection interaction feature, the second transmission contribution feature, and the third reflection global semantic aggregation feature into an intra-stream multi-scale channel interaction module to determine a third reflection interaction feature; Performing weighted fusion using the transmission flow feature, the first transmission interaction feature, the second transmission interaction feature, and the third transmission interaction feature to output an improved transmission flow feature; A weighted fusion is performed on the reflection flow feature, the first reflection interaction feature, the second reflection interaction feature, and the third reflection interaction feature to generate an improved reflection flow feature.

5. The single image reflection removal method according to claim 1, characterized in that: The processing process of the multi-scale difference feature fusion module includes: After element-by-element subtraction of the input transmission feature and reflection feature, convolution extraction and nonlinear mapping are performed through a cascade of 3×3 convolutional layers and sigmoid activation functions to output the difference position feature; The input transmission features are extracted in parallel using a 1×1 convolutional layer, a 3×3 depth-separable convolutional layer, a 5×5 depth-separable convolutional layer, and a 7×7 depth-separable convolutional layer. The features are then concatenated and fed into a 1×1 convolutional layer to generate transmission multi-scale features. After multiplying the input transmission feature by the difference position feature element by element, the multi-scale transmission feature is added element by element to determine the transmission difference feature; After extracting the input reflection features in parallel through a 1×1 convolutional layer, a 3×3 depth-separable convolutional layer, a 5×5 depth-separable convolutional layer, and a 7×7 depth-separable convolutional layer, the input is concatenated with a 1×1 convolutional layer to process the output reflection multi-scale features; After multiplying the input reflection feature by the difference position feature element by element, the multiplication is performed element by element with the reflection multi-scale feature to determine the reflection difference feature; The transmission difference feature and the reflection difference feature are grouped and spliced, and after inputting into the group convolution layer for feature processing, they are multiplied element-by-element with the difference position feature and processed by a 3×3 convolution layer to output the combined difference feature.

6. The single image reflection removal method according to claim 1, characterized in that: The processing process of the intra-stream multi-scale channel interaction module includes: After the input transmission feature or reflection feature is normalized by the normalization layer, it is projected into the in-stream value key by the input 1×1 convolution layer; After the input global semantic aggregation features are normalized by the normalization layer, they are projected into in-stream query keys and in-stream index keys respectively using a 1×1 convolutional layer, and the cosine similarity is calculated using the in-stream query keys and the in-stream index keys; Performing attention calculation based on a softmax activation function using the in-stream query key, the in-stream index key, and the in-stream value key to determine an in-stream attention feature; After calculating the dot product of the in-stream attention feature and the cosine similarity, the feature is added element-by-element to the input transmission feature or reflection feature to generate a self-adjusting feature; After concatenating the input contribution features with the self-adjusting feature channels, convolution processing is performed through a group convolution layer to output in-stream fusion features; After extracting the intra-stream fusion features in parallel through a 3×3 depthwise separable convolutional layer, a 5×5 depthwise separable convolutional layer, and a 7×7 depthwise separable convolutional layer, the features are input into a batch normalization layer for processing to determine multiple intra-stream scale features; After concatenating the in-stream scale features with the in-stream fusion feature channels, a group convolution layer is used to perform convolution fusion to generate a deep fusion feature; After the deep fusion feature is enhanced through a cascade of 1×1 convolutional layers, batch normalization layers, and GELU activation functions, it is added element-by-element with the deep fusion feature to output the transmission feature or reflection feature of the next layer.

7. The single image reflection removal method according to claim 1, characterized in that: The processing process of the inter-stream information interaction module includes: The input transmission feature and the input combined difference feature, and the input reflection feature and the combined difference feature are respectively input into the feature fusion block in pairs to perform feature fusion, and generate transmission difference fusion feature and reflection difference fusion feature respectively; Projecting the transmission difference fusion feature into a transmission index key and a transmission value key through a two-dimensional convolutional layer, and projecting the reflection difference fusion feature into a reflection index key and a reflection value key; After the reflection difference fusion feature is normalized by a layer normalization layer, it is linearly mapped to a transmission query key through a two-dimensional convolution layer and input into a pooling layer for two-dimensional upper quarter pooling to determine a transmission proxy key; After the transmission difference fusion feature is layer-normalized according to the layer normalization layer, it is linearly mapped to a reflection query key based on a two-dimensional convolution layer and input into a pooling layer for two-dimensional upper quarter pooling to determine a reflection proxy key; Performing proxy attention calculation according to the transmission index key, the transmission value key, the transmission query key and the transmission proxy key through the proxy attention layer, and outputting a transmission attention feature; Perform proxy attention calculation based on the proxy attention layer using the reflected index key, the reflected value key, the reflected query key, and the reflected proxy key, and output a reflected attention feature; After adjusting the channel semantics of the input transmission feature using a 1×1 depthwise separable convolutional layer, the input gated mixing layer performs feature fusion with the reflection attention feature to generate the reflection contribution feature; After adjusting the channel semantics of the input reflection feature using a 1×1 depthwise separable convolutional layer, the reflection feature is fused with the transmission attention feature through a gated mixing layer to generate the transmission contribution feature.

8. A single image reflection removal device, characterized in that: include: A model training module is used to train and verify the initial reflection removal model using a synthetic atlas, and then fine-tune the model based on a natural atlas to determine the target reflection removal model; The target reflection removal model includes a coarse separator, a feature encoder, a two-stream interactor and an upsampling layer; An image coarse separation module is used to input the image to be processed as the initial image of the transmission flow and the initial image of the reflection flow into the coarse separator for coarse separation, and output a coarse transmission prediction image and a coarse reflection prediction image respectively; a feature encoding module, configured to perform pyramid-type multi-scale feature enhancement on the transmission coarse prediction map and the reflection coarse prediction map respectively through the feature encoder, and correspondingly determine transmission flow features and reflection flow features; a feature interaction module, configured to use the dual-stream interactor to perform intra-stream and inter-stream feature enhancement interaction based on the transmission flow feature and the reflection flow feature, and to generate an improved transmission flow feature and an improved reflection flow feature accordingly; A prediction output module, configured to input the improved transmission flow features and the improved reflection flow features into an upsampling layer for upsampling, and output a transmission prediction map and a reflection prediction map respectively; The dual-stream interactor includes a bidirectional pooling feature aggregation module, a multi-scale difference feature fusion module, an intra-stream multi-scale channel interaction module, and an inter-stream information interaction module; The bidirectional pooling feature aggregation module includes a pooling layer, a normalization layer, a 1×1 convolution layer, a GELU activation function and a gating layer; The multi-scale difference feature fusion module includes a 1×1 convolution layer, a 3×3 convolution layer, a 3×3 depth-separable convolution layer, a 5×5 depth-separable convolution layer, a 7×7 depth-separable convolution layer, a group convolution layer and a sigmoid activation function; The intra-stream multi-scale channel interaction module includes a normalization layer, a 1×1 convolution layer, a group convolution layer, a 3×3 depth-separable convolution layer, a 5×5 depth-separable convolution layer, a 7×7 depth-separable convolution layer, a softmax activation function, a GELU activation function, and a batch normalization layer; The inter-stream information interaction module includes a feature fusion block, a layer normalization layer, a two-dimensional convolution layer, a pooling layer, a proxy attention layer, a 1×1 depthwise separable convolution layer, and a gated mixing layer.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the single image reflection removal method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the single image reflection removal method according to any one of claims 1 to 7 are implemented.

Images