Single image reflection removal method and device

Abstract

The invention discloses a single image reflection removal method and device, and relates to the technical field of image processing, and the method comprises the steps: carrying out the model training verification of an initial reflection removal model through a synthetic image set, and carrying out the model fine tuning based on a natural image set, and determining a target reflection removal model; inputting the to-be-processed image as a transmission flow initial image and a reflection flow initial image into a coarse separator for coarse separation, and correspondingly outputting a transmission coarse prediction image and a reflection coarse prediction image; a transmission flow feature and a reflection flow feature are correspondingly determined based on the transmission rough prediction map and the reflection rough prediction map through a feature encoder; performing intra-flow and inter-flow feature enhanced interaction according to the transmission flow features and the reflection flow features by adopting a double-flow interaction device, and correspondingly generating improved transmission flow features and improved reflection flow features; and respectively inputting the improved transmission flow features and the improved reflection flow features into an up-sampling layer for up-sampling, and correspondingly outputting a transmission prediction map and a reflection prediction map. The reflection removal model based on the design is beneficial for improving the reliability of image reflection removal.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular, to a single-image reflection removal method and device. Background Art

[0002] In real-world photography, the image quality is often severely affected by specular or highlight reflections. Therefore, it is necessary to perform reflection removal processing on the image to improve the image quality. In practical applications, for example, it is possible to enhance the image quality collected by the camera of intelligent driving under complex weather conditions to reduce the misjudgment probability of the intelligent driving system, reduce the specular reflection interference when shooting scenes through glass windows or other reflective surfaces to make the captured images clearer and more realistic, and remove window reflections in the license plate recognition system to improve the license plate recognition accuracy, etc.

[0003] Reflection removal is a typical ill-posed problem. The input image of the task provides limited information. Since the reflection and background information highly overlap, information loss or misjudgment is likely to occur during the removal process. Early reflection removal methods usually relied on physical devices such as rotating polarizers, and by adjusting the angle of the polarization filter, the visibility of the background scene could be changed to a certain extent, but they were prone to defects such as high cost and poor effects in complex or dynamic scenes.

[0004] With the rapid development of deep learning technology, many neural network-based image reflection removal methods have emerged. By automatically learning the relationship between the reflected image and the target image, the separation information of the reflection and transmission components in the image can be effectively extracted. For example, using a deep network and perceptual loss to solve the single-image reflection removal problem, the ERRNet model improves the image reconstruction quality by simplifying the residual block and using the semantic features extracted by the VGG-19 network, YTMT improves the reflection removal effect by promoting block-level communication between streams through a two-stream decomposition network, CoRRN enhances the robustness of the reflection removal model by combining image context and multi-scale information through a network structure sharing features, and DSRNet adopts a two-stream cascade structure and a learnable residual module (LRM) to improve the removal effect through hierarchical semantic information and module refinement. However, the existing deep learning methods still face challenges such as high computational complexity, the reflection removal accuracy being limited by the expressive power of the network structure, and it being difficult to effectively process high-resolution and complex reflection scenes, and there is a need to further improve the reliability of image reflection removal. Summary of the Invention

[0005] The present invention provides a single-image reflection removal method and device, which solves the technical problem that the existing neural network-based image reflection removal methods need to further improve the reliability of image reflection removal.

[0006] A single-image reflection removal method provided by the first aspect of the present invention includes:

[0007] After training and validating the initial reflection removal model using a synthetic atlas, the model is fine-tuned based on a natural atlas to determine the target reflection removal model; the target reflection removal model includes a coarse separator, a feature encoder, a two-stream interaction module, and an upsampling layer;

[0008] The image to be processed is input into the coarse separator as the initial transmission image and the initial reflection image respectively for coarse separation, and the coarse transmission prediction image and the coarse reflection prediction image are output correspondingly;

[0009] The pyramid multi-scale feature enhancement is respectively performed on the coarse transmission prediction image and the coarse reflection prediction image through the feature encoder to correspondingly determine the transmission stream feature and the reflection stream feature;

[0010] The two-stream interaction module is used to perform in-stream and inter-stream feature enhancement interaction according to the transmission stream feature and the reflection stream feature, and the improved transmission stream feature and the improved reflection stream feature are correspondingly generated;

[0011] The improved transmission stream feature and the improved reflection stream feature are respectively input into the upsampling layer for upsampling, and the transmission prediction image and the reflection prediction image are output correspondingly.

[0012] Optionally, the coarse separator includes an encoder, a bottleneck layer, and a decoder connected in sequence;

[0013] The encoder includes a cascade of multiple residual modules and a max pooling layer, the decoder includes a cascade of multiple residual modules and a pixel shuffle layer, and the residual modules of the encoder and the decoder are connected by skip connections;

[0014] The residual module includes a series of multiple convolutional blocks, and each convolutional block includes a 3×3 convolutional layer with residual connection, a batch normalization layer, and a ReLU activation function.

[0015] Optionally, the feature encoder includes a Swin Transformer module, a multi-scale feature extraction module, a cross-layer fusion module, and a 1×1 convolutional layer; the processing process of the feature encoder includes:

[0016] The feature encoding input image is successively encoded layer by layer using a cascade of multiple Swin Transformer modules to generate encoded features;

[0017] Each of the encoded features is respectively enhanced by the multi-scale feature extraction module for multi-scale feature enhancement to correspondingly determine enhanced features;

[0018] Based on the cross-layer fusion module, after upsampling and fusing according to each of the enhanced features, it is input into the 1×1 convolutional layer for convolutional processing to output the encoded stream feature.

[0019] Optionally, the dual-stream interactors include a bidirectional pooling feature aggregation module, a multi-scale difference feature fusion module, an intra-stream multi-scale channel interaction module, and an inter-stream information interaction module; using the dual-stream interactors to perform intra-stream and inter-stream feature enhancement interaction based on the transmitted stream features and the reflected stream features, and correspondingly generating improved transmitted stream features and improved reflected stream features, including:

[0020] The bidirectional pooling feature aggregation module respectively performs bidirectional pooling on the transmitted stream features and the reflected stream features to concentrate global semantic information, and correspondingly outputs a first transmitted global semantic aggregation feature and a first reflected global semantic aggregation feature;

[0021] The transmitted stream features and the first transmitted global semantic aggregation feature, and the reflected stream features and the first reflected global semantic aggregation feature are respectively input in pairs into the intra-stream multi-scale channel interaction module for feature interaction fusion, and correspondingly generate a first transmitted interaction feature and a first reflected interaction feature;

[0022] The multi-scale combined difference feature fusion module is used to extract difference information from the first transmitted interaction feature and the first reflected interaction feature to determine a first combined difference feature;

[0023] According to the inter-stream information interaction module, based on the proxy attention mechanism, using the first combined difference feature as a clue to guide the first transmitted interaction feature and the first reflected interaction feature to perform feature interaction, and output a first reflected contribution feature and a first transmitted contribution feature;

[0024] The first transmitted interaction feature and the first reflected interaction feature are respectively input into the bidirectional pooling feature aggregation module, and correspondingly output a second transmitted global semantic aggregation feature and a second reflected global semantic aggregation feature;

[0025] The intra-stream multi-scale channel interaction module generates a second transmitted interaction feature based on the first transmitted interaction feature, the first reflected contribution feature, and the second transmitted global semantic aggregation feature, and generates a second reflected interaction feature based on the first reflected interaction feature, the first transmitted contribution feature, and the second reflected global semantic aggregation feature;

[0026] The multi-scale combined difference feature fusion module determines a second combined difference feature according to the second transmitted interaction feature and the second reflected interaction feature;

[0027] According to the inter-stream information interaction module, based on the second transmitted interaction feature, the second reflected interaction feature, and the second combined difference feature, output a second reflected contribution feature and a second transmitted contribution feature;

[0028] Based on the second transmission interaction feature and the second reflection interaction feature respectively through the bidirectional pooling feature aggregation module, the third transmission global semantic aggregation feature and the third reflection global semantic aggregation feature are correspondingly output;

[0029] Input the second transmission interaction feature, the second reflection contribution feature and the third transmission global semantic aggregation feature into the in-stream multi-scale channel interaction module to generate a third transmission interaction feature;

[0030] Input the second reflection interaction feature, the second transmission contribution feature and the third reflection global semantic aggregation feature into the in-stream multi-scale channel interaction module to determine a third reflection interaction feature;

[0031] Perform weighted fusion on the transmission stream feature, the first transmission interaction feature, the second transmission interaction feature and the second transmission interaction feature, and output an improved transmission stream feature;

[0032] Perform weighted fusion according to the reflection stream feature, the first reflection interaction feature, the second reflection interaction feature and the third reflection interaction feature to generate an improved reflection stream feature.

[0033] Optionally, the multi-scale difference feature fusion module includes a 1×1 convolutional layer, a 3×3 convolutional layer, a 3×3 depthwise separable convolutional layer, a 5×5 depthwise separable convolutional layer, and a 7×7 depthwise separable convolutional layer, a grouped convolutional layer and a sigmoid activation function; the processing process of the multi-scale difference feature fusion module includes:

[0034] After subtracting the input transmission feature and reflection feature element by element, perform convolution extraction and non-linear mapping through a cascaded 3×3 convolutional layer and a sigmoid activation function, and output a difference position feature;

[0035] Use a 1×1 convolutional layer, a 3×3 depthwise separable convolutional layer, a 5×5 depthwise separable convolutional layer, and a 7×7 depthwise separable convolutional layer to parallelly extract features from the input transmission feature, then splice them, and input them into a 1×1 convolutional layer for processing to generate a transmission multi-scale feature;

[0036] Multiply the input transmission feature and the difference position feature element by element, and then add them to the transmission multi-scale feature element by element to determine the transmission difference feature;

[0037] Use a 1×1 convolutional layer, a 3×3 depthwise separable convolutional layer, a 5×5 depthwise separable convolutional layer, and a 7×7 depthwise separable convolutional layer to parallelly extract features from the input reflection feature, then splice and input them into a 1×1 convolutional layer for processing to output a reflection multi-scale feature;

[0038] After multiplying the input reflection feature element - by - element with the difference position feature and then adding them element - by - element to the reflection multi - scale feature, the reflection difference feature is determined.

[0039] The transmission difference feature and the reflection difference feature are grouped and concatenated, and after being input into a grouped convolution layer for feature processing, they are multiplied element - by - element with the difference position feature and processed by a 3×3 convolution layer to output a combined difference feature.

[0040] Optionally, the in - stream multi - scale channel interaction module includes a normalization layer, a 1×1 convolution layer, a grouped convolution layer, a 3×3 depth - wise separable convolution layer, a 5×5 depth - wise separable convolution layer, and a 7×7 depth - wise separable convolution layer, a softmax activation function, a GELU activation function, and a batch normalization layer; the processing process of the in - stream multi - scale channel interaction module includes:

[0041] After normalizing the input transmission feature or reflection feature through the normalization layer, it is input into a 1×1 convolution layer to project it into an in - stream value key.

[0042] After normalizing the input global semantic aggregation feature through the normalization layer, it is projected into an in - stream query key and an in - stream index key respectively by a 1×1 convolution layer, and the cosine similarity is calculated using the in - stream query key and the in - stream index key.

[0043] Based on the softmax activation function, attention calculation is performed using the in - stream query key, the in - stream index key, and the in - stream value key to determine the in - stream attention feature.

[0044] After multiplying the in - stream attention feature by the cosine similarity and then adding them element - by - element to the input transmission feature or reflection feature, a self - adjusted feature is generated.

[0045] After channel - concatenating the input contribution feature and the self - adjusted feature, convolution processing is performed through a grouped convolution layer to output an in - stream fusion feature.

[0046] After parallel feature extraction of the in - stream fusion feature by a 3×3 depth - wise separable convolution layer, a 5×5 depth - wise separable convolution layer, and a 7×7 depth - wise separable convolution layer, it is input into a batch normalization layer for processing to determine multiple in - stream scale features.

[0047] After channel - concatenating each in - stream scale feature and the in - stream fusion feature, convolution fusion is performed using a grouped convolution layer to generate a depth - fusion feature.

[0048] After feature enhancement of the depth - fusion feature through a cascaded 1×1 convolution layer, a batch normalization layer, and a GELU activation function, it is added element - by - element to the depth - fusion feature to output the next - layer transmission feature or reflection feature.

[0049] Optionally, the inter-stream information interaction module includes a feature fusion block, a layer normalization layer, a two-dimensional convolutional layer, a pooling layer, a proxy attention layer, a 1×1 depthwise separable convolutional layer, and a gated mixing layer; the processing process of the inter-stream information interaction module includes:

[0050] Pairwise input the input transmission feature and the input combined difference feature, and the input reflection feature and the combined difference feature into the feature fusion block for feature fusion, respectively generating a transmission difference fusion feature and a reflection difference fusion feature;

[0051] Project the transmission difference fusion feature into a transmission index key and a transmission value key through the two-dimensional convolutional layer, and project the reflection difference fusion feature into a reflection index key and a reflection value key;

[0052] After performing layer normalization on the reflection difference fusion feature by the layer normalization layer, linearly map it into a transmission query key through the two-dimensional convolutional layer and input it into the pooling layer for 1 / 4 pooling in two dimensions to determine the transmission proxy key;

[0053] Based on the layer normalization of the transmission difference fusion feature by the layer normalization layer, linearly map it into a reflection query key through the two-dimensional convolutional layer and input it into the pooling layer for 1 / 4 pooling in two dimensions to determine the reflection proxy key;

[0054] Perform proxy attention calculation through the proxy attention layer according to the transmission index key, the transmission value key, the transmission query key, and the transmission proxy key, and output the transmission attention feature;

[0055] Perform proxy attention calculation based on the proxy attention layer using the reflection index key, the reflection value key, the reflection query key, and the reflection proxy key, and output the reflection attention feature;

[0056] After adjusting the channel semantics of the input transmission feature using the 1×1 depthwise separable convolutional layer, input it into the gated mixing layer for feature fusion with the reflection attention feature to generate a reflection contribution feature;

[0057] After adjusting the channel semantics of the input reflection feature using the 1×1 depthwise separable convolutional layer, perform feature fusion with the transmission attention feature through the gated mixing layer to generate a transmission contribution feature.

[0058] A single-image reflection removal device provided in the second aspect of the present invention includes:

[0059] A model training module, which is used to perform model training verification on the initial reflection removal model using a synthetic atlas, and then perform model fine-tuning based on a natural atlas to determine a target reflection removal model; the target reflection removal model includes a coarse separator, a feature encoder, a two-stream interaction unit, and an upsampling layer;

[0060] An image rough separation module, configured to input the image to be processed as the initial transmission stream image and the initial reflection stream image into the rough separator respectively for rough separation, and correspondingly output a transmission rough prediction image and a reflection rough prediction image;

[0061] A feature encoding module, configured to respectively perform pyramid multi-scale feature enhancement on the transmission rough prediction image and the reflection rough prediction image through the feature encoder, and correspondingly determine the transmission stream feature and the reflection stream feature;

[0062] A feature interaction module, configured to perform in-stream and inter-stream feature enhancement interaction on the transmission stream feature and the reflection stream feature by using the dual-stream interactors, and correspondingly generate an improved transmission stream feature and an improved reflection stream feature;

[0063] A prediction output module, configured to respectively input the improved transmission stream feature and the improved reflection stream feature into an upsampling layer for upsampling, and correspondingly output a transmission prediction image and a reflection prediction image.

[0064] A computer device provided in the third aspect of the present invention includes a memory and a processor. A computer program is stored in the memory. When the computer program is executed by the processor, the processor executes the steps of the single-image reflection removal method as described in any one of the above.

[0065] A computer-readable storage medium provided in the fourth aspect of the present invention has a computer program stored thereon. When the computer program is executed, the single-image reflection removal method as described in any one of the above is implemented.

[0066] It can be seen from the above technical solutions that the present invention has the following advantages:

[0067] The above solution of the present invention provides a single-image reflection removal method, including: after training and validating an initial reflection removal model using a synthetic atlas, fine-tuning the model based on a natural atlas to determine a target reflection removal model; the target reflection removal model includes a coarse separator, a feature encoder, a two-stream interaction unit, and an upsampling layer; inputting a to-be-processed image as an initial transmission image and an initial reflection image into the coarse separator for coarse separation respectively, and correspondingly outputting a coarse transmission prediction image and a coarse reflection prediction image; respectively performing pyramid multi-scale feature enhancement on the coarse transmission prediction image and the coarse reflection prediction image through the feature encoder to correspondingly determine a transmission stream feature and a reflection stream feature; using the two-stream interaction unit to perform in-stream and inter-stream feature enhancement interaction according to the transmission stream feature and the reflection stream feature to correspondingly generate an improved transmission stream feature and an improved reflection stream feature; inputting the improved transmission stream feature and the improved reflection stream feature into the upsampling layer for upsampling respectively to correspondingly output a transmission prediction image and a reflection prediction image. Based on the above solution, the complexity to be processed in subsequent stages is effectively reduced by the coarse separator, and the robustness of the model in a complex reflection environment is effectively enhanced by performing in-stream feature interaction and inter-stream information exchange through the two-stream interaction unit. Using a two-stream structure for reflection removal helps to achieve a more accurate separation of reflection and transmission components. Therefore, the designed reflection removal model helps to improve the reliability of image reflection removal. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0069] Figure 1 It is a flowchart of the steps of a single-image reflection removal method provided by an embodiment of the present invention;

[0070] Figure 2 It is a schematic flowchart of the reflection removal model provided by an embodiment of the present invention;

[0071] Figure 3 It is a schematic structural diagram of the coarse separator provided by an embodiment of the present invention;

[0072] Figure 4 It is a schematic structural diagram of the feature encoder provided by an embodiment of the present invention;

[0073] Figure 5 It is a combined schematic diagram of the two-stream interaction unit and the upsampling layer provided by an embodiment of the present invention;

[0074] Figure 6 It is a schematic structural diagram of the bidirectional pooling feature aggregation module provided by an embodiment of the present invention;

[0075] Figure 7 Schematic diagram of the structure of the multi-scale difference feature fusion module provided by the embodiment of the present invention;

[0076] Figure 8 Schematic diagram of the structure of the in-stream multi-scale channel interaction module provided by the embodiment of the present invention;

[0077] Figure 9 Schematic diagram of the structure of the inter-stream information interaction module provided by the embodiment of the present invention;

[0078] Figure 10 Block diagram of the structure of a single-image reflection removal device provided by the embodiment of the present invention. Detailed implementation manners

[0079] The embodiment of the present invention provides a single-image reflection removal method and device, which are used to solve the technical problem that the existing neural network-based image reflection removal method needs to further improve the reliability of image reflection removal.

[0080] In order to make the objectives, features, and advantages of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the embodiments described below are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0081] Please refer to Figure 1 , Figure 1 Flowchart of the steps of a single-image reflection removal method provided by the embodiment of the present invention.

[0082] A single-image reflection removal method provided by this embodiment includes:

[0083] Step 101: After training and validating the initial reflection removal model using a synthetic atlas, perform model fine-tuning based on a natural atlas to determine a target reflection removal model; the target reflection removal model includes a coarse separator, a feature encoder, a two-stream interaction unit, and an upsampling layer.

[0084] It should be noted that in this embodiment, a model for image reflection removal is designed, which is called a reflection removal model. The composition architecture of the model includes a coarse separator, a feature encoder, a two-stream interaction unit, and an upsampling layer;

[0085] Train the initial reflection removal model. During the training process, an image dataset containing a large amount of labeled image data is required, and the image dataset is subjected to normalization processing and data preprocessing; the normalization processing includes cropping, scaling, and normalization, etc. While maintaining the aspect ratio, crop the image to remove edge interference, scale the image to a unified size of 224*224 (this size is set according to the model input), and scale the pixel values of the cropped and scaled image from [0, 255] to the range of [0, 1] to ensure that the image size meets the network input requirements; the data preprocessing includes basic preprocessing operations such as denoising and smoothing to provide clean input for model training;

[0086] The image dataset includes a synthetic atlas and a natural atlas. Each image data sample consists of a triple of a mixed image, a transmission image, and a reflection image. The difference is that the mixed image in the synthetic atlas is obtained by synthesizing an existing transmission image and a reflection image through a synthesis algorithm or model. The mixed image and the corresponding transmission image in the natural atlas are from several publicly available real-scene datasets in the field of reflection removal. The reflection image in the natural atlas can be obtained by subtracting the associated mixed image and transmission image. It can be understood that the line information and texture information in the mixed image of the synthetic atlas are much more than those in the mixed image of the natural atlas, and the distribution of reflected light in the synthetic mixed image is not the same as that in the natural mixed image; in one implementation, the formula for synthesizing the mixed image can be expressed as , is the first weight coefficient, is the second weight coefficient, is the virtual and real distribution range of the reflection area, is the transmission image, is the reflection image, is the mixed image;

[0087] In this embodiment, the initial reflection removal model is first trained and verified using a synthetic atlas to determine an intermediate reflection removal model: the synthetic atlas is divided into a synthetic training set and a synthetic validation set. The initial reflection removal model is iteratively trained using the synthetic training set with the goal of minimizing the loss function value, and the synthetic validation set is used for model validation and evaluation to determine the intermediate reflection removal model. Thus, the weight of the round with the optimal effect in the synthetic atlas stage is selected as the final model weight, and the reflection removal effect and generalization ability of the model are tested using the validation set for quantitative evaluation. Then, based on a natural atlas, model fine-tuning is performed to determine the target reflection removal model: similar to the process of the synthetic atlas, the natural atlas is divided into a natural training set and a natural validation set. The initial reflection removal model is iteratively trained using the natural training set with the goal of minimizing the loss function value, and the natural validation set is used for model validation and evaluation to determine the target reflection removal model. This two-stage training method, on the one hand, is that it is difficult to obtain natural atlases, while synthetic atlases can be synthesized from any two images into a mixed image, and it is easier to obtain a large number of training sets containing mixed images, corresponding transmission images, and corresponding reflection images. Therefore, most of the data in the composed image training set should come from synthetic atlases. On the other hand, after using the synthetic atlas to train the model to have some separation ability in texture and lines, then using the natural atlas to train the model to learn the distribution of natural light in the real world, the processing efficiency and processing effect will be better.

[0088] In a specific implementation manner of this embodiment, the overall loss function for calculating the loss function value combines SSIM loss, PSNR loss, and perceptual loss, and can improve the separation accuracy of reflection and transmission images while retaining the image structure and visual quality; Figure 2 As can be seen, the reflection removal model takes the mixed image as the model input and the transmission image and the reflection image as the output. Therefore, the overall loss function of this embodiment includes the sum of the total transmission flow loss function and the reflection flow loss function. The predicted transmission map output by the model training and the associated real transmission map are input into the total transmission flow loss function for calculation, and the predicted reflection map output by the model training and the associated real reflection map are input into the reflection flow loss function for calculation. The calculation formulas for the total transmission flow loss function or the reflection flow loss function can be referred to: , is the total loss function, is the structural similarity loss, is the structural weight coefficient, is the peak signal-to-noise ratio loss, is the signal-to-noise weight coefficient, is the perceptual loss, is the perceptual weight coefficient, and the calculation process of each loss can be specifically referred to the prior art; taking the transmission flow as an example, the structural similarity loss is used to measure the structural difference between the image after reflection removal and the real transmission image, the peak signal-to-noise ratio loss is used to ensure the similarity between the removed image and the real transmission image at the pixel level, and the perceptual loss is used to compare the semantic representations of the features between the generated image and the target image at different levels by using mainstream image model architectures (such as Vgg16, Vgg19 or MobileNet, etc.).

[0089] In a specific implementation manner of this embodiment, in order to ensure that the reflection removal model can operate efficiently on resource-constrained embedded devices, after the model training is completed, model quantization and pruning operations for model lightweighting are adopted; after estimating the size and computational amount of the lightweight model, the trained reflection removal model is converted from the standard TensorFlow or PyTorch format to a format compatible with embedded devices through frameworks such as TensorFlow Lite and ONNX, and the inference process of the model is optimized by using hardware acceleration (such as GPU, TPU, etc.) through the acceleration library of TensorFlow Lite or frameworks such as OpenVINO to further improve the running speed and efficiency. Finally, the lightweight model can be deployed to devices such as NVIDIA Jetson Nano or other devices with GPU acceleration functions, and images can be captured in real time through an integrated camera module and input into the target reflection removal model for processing. The processed images can be directly used for subsequent applications, such as license plate recognition and reflection removal in camera photography.

[0090] Step 102: Input the image to be processed into the coarse separator as the initial transmission flow image and the initial reflection flow image respectively, and correspondingly output the coarse transmission prediction image and the coarse reflection prediction image.

[0091] It should be noted that when receiving the image to be processed (in the form of a mixed image), the image to be processed is copied into two, which are input into the coarse separator as the initial transmission flow image and the initial reflection flow image respectively, and then the coarsely separated coarse transmission prediction image and coarse reflection prediction image are obtained, initially establishing a two-stream architecture, aiming to reduce the ill-posedness of the input image for the subsequent separation work of the network.

[0092] In a specific implementation manner of this embodiment, the coarse separator includes an encoder, a bottleneck layer and a decoder connected in sequence;

[0093] The encoder includes a plurality of cascaded residual modules and a max pooling layer, the decoder includes a plurality of cascaded residual modules and a pixel shuffle layer, and the residual modules of the encoder and the decoder are connected by skip connections;

[0094] The residual module includes multiple cascaded convolutional blocks, and each convolutional block includes a 3×3 convolutional layer with residual connection, a batch normalization layer, and a ReLU activation function.

[0095] It should be noted that, in this embodiment, the structure of the coarse separator is as Figure 3 shown. The coarse separator uses the U-Net structure as the basic network. The encoder and the decoder are connected in series through a bottleneck layer in the middle. There are corresponding multi-layer residual modules in the encoder and the decoder. Each residual module is specifically composed of two cascaded convolutional blocks, and each convolutional block has a structure of "3*3Conv-BatchNorm-ReLU". Residual connections are used between the blocks. The processing process of each convolutional block can be expressed by the formula: , is the output of the convolutional block, is the input of the convolutional block, is the ReLU activation function, is the batch normalization layer, is the 3×3 convolutional layer.

[0096] Step 103: Respectively perform pyramid multi-scale feature enhancement on the transmitted coarse prediction map and the reflected coarse prediction map through the feature encoder, and correspondingly determine the transmitted flow feature and the reflected flow feature.

[0097] It should be noted that the feature encoder designed in this embodiment combines the pyramid mechanism and the multi-scale feature enhancement mechanism. After inputting the transmitted coarse prediction map and the reflected coarse prediction map into the feature encoder for feature processing, the transmitted flow feature and the reflected flow feature are correspondingly output. The parameters of one encoder are shared on the transmitted side and the reflected side, which is convenient for projecting the images of both streams into the same space.

[0098] In a specific implementation manner of this embodiment, the feature encoder includes a Swim Transformer module, a multi-scale feature extraction module, a cross-layer fusion module, and a 1×1 convolutional layer; the processing process of the feature encoder includes:

[0099] S11: Layer-by-layer perform feature encoding on the feature encoding input map using multiple cascaded Swim Transformer modules, and sequentially generate encoded features;

[0100] S12: Respectively perform multi-scale feature enhancement on each encoded feature through the multi-scale feature extraction module, and correspondingly determine the enhanced features;

[0101] S13: Based on the cross-layer fusion module, perform upsampling and fusion according to each enhanced feature, and then input it into the 1×1 convolutional layer for convolution processing to output the encoded flow feature.

[0102] It should be noted that the structure of the feature encoder in this embodiment is as Figure 4 shown. A pre-trained SwimTransformer module is used as the basic network to construct a 5-layer pyramid super-column feature. The reason for using Swim Transformer as the encoder is that it can effectively capture the long-range dependence information and global semantics of the reflected image in the mixed image, and at the same time, the number of parameters is less than that of the classic Vision Transformer. Each layer of the Swim Transformer module in the feature encoder is connected with a multi-scale feature extraction module and a cross-layer fusion module. The encoded features output by the Swim Transformer model at each level are connected to the multi-scale feature extraction module, and the enhanced features output from the multi-scale feature extraction module are fed into the cross-layer fusion module, and up-sampled feature fusion is performed in the order from deep to shallow, and finally input into a 1×1 convolutional layer for convolution processing. When the transmitted rough prediction map and the reflected rough prediction map of the two-stream architecture are input into the feature encoder, the corresponding output encoded stream features include the transmitted stream features and the reflected stream features.

[0103] In a more specific implementation manner of this embodiment, the multi-scale feature extraction module includes a 1×1 convolutional layer, a 3×3 depthwise separable convolutional layer, a 5×5 depthwise separable convolutional layer, and a 7×7 depthwise separable convolutional layer; the processing process of the multi-scale feature extraction module includes:

[0104] S1201: After parallelly extracting features from the input encoded features using a 3×3 depthwise separable convolutional layer, a 5×5 depthwise separable convolutional layer, and a 7×7 depthwise separable convolutional layer, they are concatenated with the input encoded features to generate scale features;

[0105] S1202: The scale features are convolved through a 1×1 convolutional layer to output the corresponding enhanced features.

[0106] It should be noted that as Figure 4 shown, the structure of the multi-scale feature extraction module is multiple parallel convolutional layers of different scales (3×3DWConv, 5×5DWConv, and 7×7DWConv), and the outputs of each scale convolutional layer are concatenated and then fused through a 1×1 Conv to restore the number of channels.

[0107] In a more specific implementation manner of this embodiment, the cross-layer fusion module includes an upsampling layer and a depthwise separable convolutional layer; sub-step S13 includes:

[0108] S1301: The enhanced features of the deepest layer are input into the upsampling layer of the cross-layer fusion module of the deepest layer for upsampling, and after being concatenated with the enhanced features of the previous layer, they are convolved using a depthwise separable convolutional layer to generate the first fusion feature;

[0109] S1302. After upsampling the first fusion feature in the upsampling layer of the cross-layer fusion module in the previous layer, concatenate it with the enhanced feature in the previous layer and perform convolution processing using a depthwise separable convolutional layer to determine the new first fusion feature, and input it into the cross-layer fusion module in the previous layer for upsampling, concatenation, and convolution processing until it is input into a 1×1 convolutional layer for feature processing to output the encoded stream feature.

[0110] It should be noted that the cross-layer fusion module consists of "Upsample-Concat-DWConv". Among them, the upsampling layer (Upsample) first widens the features in the deep layer by 2 times the number of channels through a 1×1 convolution and then upsamples by a factor of 2. Without affecting the original number of channels, it reaches the same size as the features in the previous layer. Then, the features of the two layers are concatenated and fused through a depthwise separable convolutional layer (DWConv) and used as the deep input of the cross-layer fusion module in the upper layer. Finally, the fused features with the same size as the first layer of the Swim Transformer module are used as the output of the feature encoder.

[0111] Step 104. Use a two-stream interaction unit to perform intra-stream and inter-stream feature enhancement interaction based on the transmitted stream feature and the reflected stream feature, and correspondingly generate an improved transmitted stream feature and an improved reflected stream feature.

[0112] It should be noted that the transmitted stream feature and the reflected stream feature output after encoding will find the features required by the current stream in the two-stream interaction unit and output these features to it to achieve inter-stream feature interaction. At the same time, within the current stream, according to the information obtained by its own feature enhancement, combined with the information obtained from inter-stream feature interaction, intra-stream feature interaction is performed within the current stream to enhance the feature expression ability, and finally an improved transmitted stream feature and an improved reflected stream feature are output.

[0113] In a specific implementation manner of this embodiment, the two-stream interaction unit includes a bidirectional pooling feature aggregation module (BPFAM), a multi-scale difference feature fusion module (MDFM), an intra-stream multi-scale channel interaction module (MSCI), and an inter-stream information interaction module (ISII); Step 104 includes the following sub-steps:

[0114] S21. Respectively perform bidirectional pooling on the transmitted stream feature and the reflected stream feature to concentrate the global semantic information through bidirectional pooling, and correspondingly output a first transmitted global semantic aggregation feature and a first reflected global semantic aggregation feature ;

[0115] S22. The transmitted stream feature With the first transmission global semantic aggregation feature , the reflection flow feature The first reflection global semantic aggregation feature Are respectively input in pairs into the in-flow multi-scale channel interaction module for feature interaction and fusion, and correspondingly generate the first transmission interaction feature And the first reflection interaction feature ;

[0116] S23. Use the multi-scale combined difference feature fusion module to extract the difference information from the first transmission interaction feature And the first reflection interaction feature To determine the first combined difference feature ;

[0117] S24. According to the inter-flow information interaction module based on the proxy attention mechanism, use the first combined difference feature As a clue to guide the first transmission interaction feature And the first reflection interaction feature To perform feature interaction and output the first reflection contribution feature And the first transmission contribution feature ;

[0118] S25. Input the first transmission interaction feature And the first reflection interaction feature Into the bidirectional pooling feature aggregation module respectively, and correspondingly output the second transmission global semantic aggregation feature And the second reflection global semantic aggregation feature ;

[0119] S26. Through the in-flow multi-scale channel interaction module, based on the first transmission interaction feature , the first reflection contribution feature And the second transmission global semantic aggregation feature Generate the second transmission interaction feature , and based on the first reflection interaction feature , the first transmission contribution feature And the second reflection global semantic aggregation feature Generate the second reflection interaction feature ;

[0120] S27. Use the multi-scale combined difference feature fusion module to determine the second combined difference feature according to the second transmission interaction feature And the second reflection interaction feature To determine the second combined difference feature ;

[0121] S28. According to the inter-flow information interaction module based on the second transmission interaction feature , the second reflection interaction feature and the second combined difference feature , output the second reflection contribution feature and the second transmission contribution feature ;

[0122] S29. Through the bidirectional pooling feature aggregation module, respectively based on the second transmission interaction feature and the second reflection interaction feature , correspondingly output the third transmission global semantic aggregation feature and the third reflection global semantic aggregation feature ;

[0123] S210. Input the second transmission interaction feature , the second reflection contribution feature and the third transmission global semantic aggregation feature into the in-stream multi-scale channel interaction module to generate the third transmission interaction feature ;

[0124] S211. Input the second reflection interaction feature , the second transmission contribution feature and the third reflection global semantic aggregation feature into the in-stream multi-scale channel interaction module to determine the third reflection interaction feature ;

[0125] S212. Use the transmission flow feature , the first transmission interaction feature , the second transmission interaction feature and the second transmission interaction feature to perform weighted fusion and output the improved transmission flow feature ;

[0126] S213. According to the reflection flow feature , the first reflection interaction feature , the second reflection interaction feature and the third reflection interaction feature to perform weighted fusion and generate the improved reflection flow feature .

[0127] It should be noted that, as Figure 5As shown, the encoded reflection features and transmission features will interact and fuse in the dual-stream interactors, mainly through two modules: the intra-stream multi-scale channel interaction module (MSCI) and the inter-stream information interaction module (ISII). The MSCI is responsible for fusing the useful information from the features of the other stream with the features of its own stream and enhancing the features unique to its own stream. The ISII is responsible for interacting the two streams, finding the features required by the other stream in its own stream in the way of proxy attention, and outputting these features to it. The dual-stream interaction learning method based on proxy attention has a low computational complexity while improving the accuracy and efficiency of image reflection removal, and can be effectively deployed on embedded devices. Especially in the application of high-resolution image processing and complex scenes, it can also run efficiently on devices with limited resources, having stronger practicality.

[0128] Taking the transmission stream as the main stream as an example: After the features of the transmission stream are output from the feature encoder as the transmission stream features and pass through the MSCI for the first time, they first pass through the bi-directional pooling feature aggregation module (BPFAM) to condense the global semantic information of the features to obtain the first transmission global semantic aggregation feature and input this vector together with into the MSCI to obtain In the same way, is obtained in the reflection stream and then enters the ISII. In the ISII, first, and are processed by the multi-scale difference feature fusion module (MDFM) to obtain the difference information vector of these two features. Then, using this difference vector as a clue, fully interacts to obtain and respectively, that is, the features contributed by the reflection stream to the transmission stream and the features contributed by the transmission stream to the reflection stream In the MSCI of the next layer of the transmission stream, , , are collected and processed to obtain After 3 times of MSCI and 2 times of ISII, the processed features and are obtained for the transmission stream and the reflection stream respectively. The dual-stream interaction stage has 3 layers. In each of the first two layers, each of the transmission stream and the reflection stream has an MSCI module, sharing one ISII module. The last layer only has an MSCI module without an ISII module. The input of each layer of the two streams is connected with the output of the third layer after residual connection and fused to obtain the final output. That is, in the transmission stream, , , , The improved transmission flow features are obtained after weighted fusion , which can be expressed by the formula: , and in the reflection flow, it is expressed as the improved reflection flow features , where , is the feature index layer in the two-stream interaction unit, is the fusion weight, is the transmission feature of the transmission flow in the two-stream interaction unit, is the reflection feature of the reflection flow in the two-stream interaction unit.

[0129] In a more specific implementation manner of this embodiment, the bidirectional pooling feature aggregation module (BPFAM) includes a pooling layer, a normalization layer, a 1×1 convolutional layer, a GELU activation function, and a gating layer; the processing process of the bidirectional pooling feature aggregation module includes:

[0130] S31. Respectively perform pooling on the input transmission feature or reflection feature along the width direction and the length direction through the pooling layer to generate a width pooling feature and a length pooling feature;

[0131] S32. Sequentially perform normalization, convolutional extraction, and non-linear mapping on the width pooling feature by using a cascaded normalization layer, a 1×1 convolutional layer, and a GELU activation function to determine the width semantic feature;

[0132] S33. After inputting the width pooling feature into the 1×1 convolutional layer for feature processing, add it to the width semantic feature element by element and output the width semantic aggregation feature;

[0133] S34. Perform normalization, convolutional extraction, and non-linear mapping on the length pooling feature by using a cascaded normalization layer, a 1×1 convolutional layer, and a GELU activation function to generate the length semantic feature;

[0134] S35. After inputting the length pooling feature into the 1×1 convolutional layer for feature processing, add it to the length semantic feature to determine the length semantic aggregation feature;

[0135] S36. Adaptively fuse the width semantic aggregation feature and the length semantic aggregation feature through the gating layer and output the global semantic aggregation feature.

[0136] It should be noted that the input of the MSCI module requires the use of the BPFAM module. As Figure 6 shown, the BPFAM module receives the transmission feature of the transmission flow ( , and ) or the reflection feature of the reflection flow ( , and ),(h-pooling) along the length direction and (w-pooling) along the width direction are then performed respectively. The features obtained by performing pooling in different directions are respectively subjected to channel feature semantic adjustment through the feature extraction blocks with residual connections. The feature extraction block includes a cascaded normalization layer, a 1×1 convolutional layer, and a GELU activation function with a 1×1 convolutional layer residual connection, so as to obtain the semantic aggregation features in this direction. Finally, the semantic aggregation features in the two directions are adaptively fused through a gating mechanism to obtain the global semantic aggregation features at the channel level.

[0137] It can be understood that the transmission features in this embodiment can refer to the features as the transmission stream in the two-stream interaction device 、 、 or ,and the reflection features can refer to the features as the reflection stream in the two-stream interaction device 、 、 or 。

[0138] In a more specific implementation manner of this embodiment, the multi-scale difference feature fusion module (MDFM) includes a 1×1 convolutional layer, a 3×3 convolutional layer, a 3×3 depthwise separable convolutional layer, a 5×5 depthwise separable convolutional layer, a 7×7 depthwise separable convolutional layer, a grouped convolutional layer, and a sigmoid activation function; the processing process of the multi-scale difference feature fusion module includes:

[0139] S41. After subtracting the input transmission features and reflection features element by element, convolution extraction and nonlinear mapping are performed through a cascaded 3×3 convolutional layer and a sigmoid activation function, and the differential position features are output;

[0140] S42. The input transmission features are parallelly extracted by using a 1×1 convolutional layer, a 3×3 depthwise separable convolutional layer, a 5×5 depthwise separable convolutional layer, and a 7×7 depthwise separable convolutional layer, then spliced, and input into a 1×1 convolutional layer for processing to generate transmission multi-scale features;

[0141] S43. After multiplying the input transmission features and the differential position features element by element, they are added to the transmission multi-scale features element by element to determine the transmission differential features;

[0142] S44. The input reflection features are parallelly extracted by using a 1×1 convolutional layer, a 3×3 depthwise separable convolutional layer, a 5×5 depthwise separable convolutional layer, and a 7×7 depthwise separable convolutional layer, then spliced and input into a 1×1 convolutional layer for processing to output reflection multi-scale features;

[0143] S45. After multiplying the input reflection feature and the difference position feature element by element, add them to the reflection multi-scale feature element by element to determine the reflection difference feature;

[0144] S46. Group and splice the transmission difference feature and the reflection difference feature, input them into the grouped convolution layer for feature processing, then multiply them by the difference position feature element by element and use a 3×3 convolution layer to process and output the combined difference feature.

[0145] It should be noted that the ISII module needs to use the MDFM module to find the difference information between the two streams as a clue to guide the information interaction between the streams; as Figure 7 shown, in the MDFM, first input the transmission feature (referred to here as or ) of the transmission stream and the reflection feature (referred to here as or ) of the reflection stream through element-wise subtraction, Conv and sigmoid operations to initially extract the spatial distribution of the difference positions on the image. This process is expressed by the formula , is the feature index layer in the two-stream interaction unit, is the difference position feature, is the 3×3 convolution layer, is the sigmoid activation function; then, perform multi-scale feature fusion on the features of the two streams through convolution kernels of different scales (1×1Conv, 3×3DWConv, 5×5DWConv, and 7×7DWConv) respectively, so that more attention can be paid to the impact brought by the spatial difference during the subsequent difference information fusion. Taking as an example, this process can be expressed by the formula for the transmission multi-scale feature , is the 1×1 convolution layer, is the 3×3 depthwise separable convolution layer, is the 5×5 depthwise separable convolution layer, is the 7×7 depthwise separable convolution layer, is the splicing; then multiply the difference position feature extracted in the first step by the original input feature and add it to the vector obtained through multi-scale feature extraction. This can strengthen the intensity of the difference position feature pixels on the basis of the original input and highlight the difference information; finally, group and splice the features of the two streams that highlight the difference information, fuse them using grouped convolution, multiply them by the difference position feature, and process them through a 3×3 convolution layer to output the combined difference feature, thereby obtaining the common difference feature of the two streams.

[0146] In a more specific implementation manner of this embodiment, the in-stream multi-scale channel interaction module (MSCI) includes a normalization layer, a 1×1 convolutional layer, a grouped convolutional layer, a 3×3 depthwise separable convolutional layer, a 5×5 depthwise separable convolutional layer, and a 7×7 depthwise separable convolutional layer, a softmax activation function, a GELU activation function, and a batch normalization layer; the processing process of the in-stream multi-scale channel interaction module includes:

[0147] S51. After normalizing the input transmission feature or reflection feature through the normalization layer, project it into an in-stream value key through the 1×1 convolutional layer;

[0148] S52. After normalizing the input global semantic aggregation feature through the normalization layer, project it into an in-stream query key and an in-stream index key respectively through the 1×1 convolutional layer, and calculate the cosine similarity using the in-stream query key and the in-stream index key;

[0149] S53. Based on the softmax activation function, perform attention calculation using the in-stream query key, the in-stream index key, and the in-stream value key to determine the in-stream attention feature;

[0150] S54. After multiplying the in-stream attention feature and the cosine similarity and then adding them element-wise to the input transmission feature or reflection feature, generate a self-adjusted feature;

[0151] S55. After concatenating the input contribution feature and the self-adjusted feature in channels, perform convolutional processing through the grouped convolutional layer to output an in-stream fusion feature;

[0152] S56. After parallel feature extraction of the in-stream fusion feature through the 3×3 depthwise separable convolutional layer, the 5×5 depthwise separable convolutional layer, and the 7×7 depthwise separable convolutional layer, input it into the batch normalization layer for processing to determine multiple in-stream scale features;

[0153] S57. After concatenating each in-stream scale feature and the in-stream fusion feature in channels, perform convolutional fusion using the grouped convolutional layer to generate a depth fusion feature;

[0154] S58. After enhancing the depth fusion feature through the cascaded 1×1 convolutional layer, the batch normalization layer, and the GELU activation function, add it element-wise to the depth fusion feature to output the transmission feature or reflection feature of the next layer.

[0155] It should be noted that, as Figure 8 shown, the in-stream multi-scale channel interaction module generally accepts three inputs. Taking the transmission stream as an example, the inputs include the transmission feature , the global semantic aggregation feature , and the contribution feature , the in-stream multi-scale channel interaction module can extract image features from multiple scales, thus enhancing the model's sensitivity and robustness to information at different scales;

[0156] The obtained global semantic aggregation feature condenses the information in the space into a value, but the information between channels has not been fully communicated. Therefore, taking this global semantic aggregation feature as a projection to obtain the query key Q and the index key K, and projecting it as the value key V to perform attention calculation. A weighted residual connection is also used, that is, the process of adjusting the feature by attention assigns weights according to the overall cosine similarity between the query key Q and the index key K, aiming to adjust the information between channels according to the similarity. The output of this process is the self-adjusted feature of the transmission flow , is the cosine inner product, is the temperature control, generally the square root of the K dimension, is the softmax activation function, is the transpose of the index key;

[0157] However, only adjusts itself according to its own information, so it can fuse the information from another flow to better recover the transmission map from the mixed map; first, and are subjected to channel-level Concat and grouped convolutions to initially and briefly fuse these two features. Next, multi-scale feature extraction is performed through depth convolution kernels of different scales (3×3DWConv, 5×5DWConv, and 7×7DWConv), and then through channel-level skip Concat and grouped convolutions to fuse the features of different scales. Finally, a feature enhancement block with a residual connection (1×1Conv-BatchNorm-GELU) is used to refine the features just extracted. The processing process of the feature enhancement block can be expressed as the flow feature of the next layer of the transmission flow , is the feature index layer in the two-stream interaction module, is the deeply fused feature, is the GELU activation function, is the 1×1 convolutional layer, is the batch normalization layer.

[0158] It can be understood that Figure 8 the "R" in it refers to the reshape operation, that is, changing the feature shape correspondingly to meet the shape required for matrix multiplication, which can be set according to the corresponding feature processing.

[0159] In a more specific implementation manner of this embodiment, the Inter-stream Information Interaction Module (ISII) includes a feature fusion block, a layer normalization layer, a two-dimensional convolutional layer, a pooling layer, a proxy attention layer, a 1×1 depthwise separable convolutional layer, and a gated mixing layer; the processing process of the Inter-stream Information Interaction Module includes:

[0160] Pairwise input the input transmission feature and the input combined difference feature, and the input reflection feature and the combined difference feature into the feature fusion block for feature fusion, and correspondingly generate the transmission difference fusion feature and the reflection difference fusion feature ;

[0161] Project the transmission difference fusion feature into a transmission index key and a transmission value key through the two-dimensional convolutional layer, and project the reflection difference fusion feature into a reflection index key and a reflection value key;

[0162] After performing layer normalization on the reflection difference fusion feature using the layer normalization layer, linearly map it into a transmission query key through the two-dimensional convolutional layer and input it into the pooling layer for 1 / 4 pooling in two dimensions to determine the transmission proxy key;

[0163] Based on the layer normalization of the transmission difference fusion feature performed by the layer normalization layer, linearly map it into a reflection query key based on the two-dimensional convolutional layer and input it into the pooling layer for 1 / 4 pooling in two dimensions to determine the reflection proxy key;

[0164] Perform proxy attention calculation through the proxy attention layer according to the transmission index key, transmission value key, transmission query key, and transmission proxy key, and output the transmission attention feature;

[0165] Based on the proxy attention layer, perform proxy attention calculation using the reflection index key, reflection value key, reflection query key, and reflection proxy key, and output the reflection attention feature;

[0166] After adjusting the channel semantics of the input transmission feature using the 1×1 depthwise separable convolutional layer, input it into the gated mixing layer to perform feature fusion with the reflection attention feature to generate the reflection contribution feature;

[0167] After adjusting the channel semantics of the input reflection feature using the 1×1 depthwise separable convolutional layer, perform feature fusion with the transmission attention feature through the gated mixing layer to generate the transmission contribution feature.

[0168] It should be noted that at this stage, as Figure 9 shown, the transmission feature of the transmission stream and that of the reflection stream will be in the combined difference feature Under the guidance of the proxy attention mechanism for interaction, taking the transmission of features from the transmission flow to the reflection flow as an example: first, and are fused through a feature fusion block (fusion) to obtain ), and in the same way, is obtained. After concatenating the two features, the number of channels is the sum of the number of channels of the two features. While reducing the number of channels through a 1×1 convolution, information exchange between channels can be achieved; After linear transformation, the index key K and the value key V are obtained. Then, is used as the Q key after layer normalization and linear mapping operations, and the proxy key Agent is obtained after 2D 1 / 4 pooling of the Q key feature. It is the core of the proxy attention. Different from the ordinary self-attention mechanism, the ordinary self-attention mechanism can be expressed by the formula , is the attention mechanism, is the attention output, while the proxy attention consists of two Softmax attention operations, namely proxy aggregation and proxy broadcasting, and can be expressed by the formula , that is, first regard the proxy token A as the query key, calculate the attention between A, K, and V to aggregate the proxy features from all values V , then regard A as the index key, as the value key, and together with the query key Q in the second attention calculation, broadcast the global information from the proxy features to each query token and finally obtain the output . The entire process of proxy attention will add the proxy bias, that is, the first proxy bias and the second proxy bias . Therefore, the complete formula is: , is the transpose of the proxy key. Here, is querying which information is useful for itself to . Finally, after the output residual connection that adjusts the channel semantics of the original input through a 1×1 DWConv, it is fused with through the gating mechanism to form , , TR refers to T to R, that is, the features of T to R, representing the information transmitted from the transmission flow to the reflection flow. The Gate mixing operation can be expressed as: ; After this operation is symmetric, is obtained, representing the information transmitted from the reflection flow to the transmission flow.

[0169] It can be understood thatFigure 9 The "P" in Figure 9 refers to four - to - one pooling, and the general characteristic shape is a square * number of channels. Quarter pooling can select representative pixels with relatively small errors. If we want to maintain the square feature, except for quarter pooling, usually only sixteenth pooling remains, which is likely to cause an increase in errors. Pooling is to downsample the feature Q into feature A, so that the computational cost will be reduced significantly when performing the attention mechanism subsequently. For example, if the shape of Q is (256, 256, C), the shape of K is (256, 256, C), and the shape of A after pooling Q is (128, 128, C), the computational cost of A * K + A * Q in this embodiment is 128 * 128 * 256 * 256 * 2, while the computational cost of the classic attention mechanism Q * K is 256 * 256 * 256 * 256. It can be seen that the method of this embodiment is computationally less by half just in terms of computational cost.

[0170] Step 105: Input the improved transmission flow feature and the improved reflection flow feature into the upsampling layer for upsampling respectively, and correspondingly output the transmission prediction map and the reflection prediction map.

[0171] Exemplarily, for the above - mentioned method embodiments, they can be deployed to the following scenarios:

[0172] Scenario 1: In the intelligent driving system at night, the camera in the vehicle may obtain images reflected by the front windshield under the influence of the light source inside the vehicle, which affects the judgment of the intelligent driving system. In the vehicle system, by implanting the above - mentioned target reflection removal model into imaging embedded devices such as cameras to execute steps 102 - 105, the window reflection can be processed in real time to ensure that the judgment of the intelligent driving system is not affected by the reflected image.

[0173] Scenario 2: When the visual obstacle avoidance function equipped on the drone is used in strong daylight, especially when shooting areas such as lakes, water surfaces or areas with glass windows, the problem of specular reflection often occurs. The reflected images of the reflections in these scenarios will affect the accurate judgment of the distance of obstacles by the visual obstacle avoidance function, resulting in unnecessary emergency braking and affecting the user experience. By implanting the above - mentioned target reflection removal model into imaging embedded devices such as cameras on the drone to execute steps 102 - 105, the captured images can be processed in real time and the reflection can be removed to provide clear image data to ensure the normal operation of the visual obstacle avoidance function.

[0174] In the embodiments of the present invention, the complexity that needs to be processed in the subsequent stage is effectively reduced by the coarse separator, the computational burden is significantly reduced, and the training and inference processes are accelerated. The in - flow feature interaction and inter - flow information exchange are carried out by the two - stream interaction unit, which effectively enhances the robustness of the model in complex reflection environments. Using the two - stream structure for reflection removal helps to achieve a more accurate separation of the reflection and transmission components. Therefore, overall, the designed reflection removal model helps to improve the reliability of image reflection removal.

[0175] Please refer to Figure 10 , Figure 10 , which is a structural block diagram of a single-image reflection removal device provided by an embodiment of the present invention.

[0176] A single-image reflection removal device provided by the present invention includes:

[0177] A model training module 1001, which is used to train and verify an initial reflection removal model using a synthetic atlas, and then fine-tune the model based on a natural atlas to determine a target reflection removal model; the target reflection removal model includes a coarse separator, a feature encoder, a two-stream interaction module, and an upsampling layer;

[0178] An image coarse separation module 1002, which is used to input a to-be-processed image as an initial transmission stream image and an initial reflection stream image into the coarse separator for coarse separation respectively, and correspondingly output a transmission coarse prediction image and a reflection coarse prediction image;

[0179] A feature encoding module 1003, which is used to perform pyramid multi-scale feature enhancement on the transmission coarse prediction image and the reflection coarse prediction image respectively through the feature encoder, and correspondingly determine a transmission stream feature and a reflection stream feature;

[0180] A feature interaction module 1004, which is used to perform in-stream and inter-stream feature enhancement interaction on the transmission stream feature and the reflection stream feature by using the two-stream interaction module, and correspondingly generate an improved transmission stream feature and an improved reflection stream feature;

[0181] A prediction output module 1005, which is used to input the improved transmission stream feature and the improved reflection stream feature into the upsampling layer for upsampling respectively, and correspondingly output a transmission prediction image and a reflection prediction image.

[0182] Further, the coarse separator includes an encoder, a bottleneck layer, and a decoder connected in sequence;

[0183] The encoder includes a plurality of cascaded residual modules and a max pooling layer, the decoder includes a plurality of cascaded residual modules and a pixel shuffle layer, and the residual modules of the encoder and the decoder are correspondingly skip-connected;

[0184] The residual module includes a plurality of cascaded convolution blocks, and each convolution block includes a 3×3 convolution layer with a residual connection, a batch normalization layer, and a ReLU activation function.

[0185] Further, the feature encoder includes a Swim Transformer module, a multi-scale feature extraction module, a cross-layer fusion module, and a 1×1 convolution layer; the processing process of the feature encoder includes:

[0186] Performing feature encoding on the feature encoding input image layer by layer using a plurality of cascaded Swim Transformer modules to generate encoded features in sequence;

[0187] Each coding feature is subjected to multi-scale feature enhancement through a multi-scale feature extraction module, and corresponding enhanced features are determined.

[0188] After upsampling and fusing based on each enhanced feature by a cross-layer fusion module, it is input into a 1×1 convolutional layer for convolutional processing, and coding stream features are output.

[0189] Furthermore, the dual-stream interactors include a bidirectional pooling feature aggregation module, a multi-scale difference feature fusion module, an intra-stream multi-scale channel interaction module, and an inter-stream information interaction module; the feature interaction module 1004 is specifically used for:

[0190] The bidirectional pooling feature aggregation module respectively performs bidirectional pooling on the transmitted stream feature and the reflected stream feature to concentrate global semantic information, and correspondingly outputs a first transmitted global semantic aggregation feature and a first reflected global semantic aggregation feature;

[0191] The transmitted stream feature and the first transmitted global semantic aggregation feature, and the reflected stream feature and the first reflected global semantic aggregation feature are respectively input into the intra-stream multi-scale channel interaction module in pairs for feature interaction and fusion, and correspondingly generate a first transmitted interaction feature and a first reflected interaction feature;

[0192] The multi-scale combined difference feature fusion module is used to extract difference information from the first transmitted interaction feature and the first reflected interaction feature to determine a first combined difference feature;

[0193] According to the inter-stream information interaction module, based on the proxy attention mechanism, using the first combined difference feature as a clue to guide the feature interaction between the first transmitted interaction feature and the first reflected interaction feature, and output a first reflected contribution feature and a first transmitted contribution feature;

[0194] The first transmitted interaction feature and the first reflected interaction feature are respectively input into the bidirectional pooling feature aggregation module, and correspondingly output a second transmitted global semantic aggregation feature and a second reflected global semantic aggregation feature;

[0195] The intra-stream multi-scale channel interaction module generates a second transmitted interaction feature based on the first transmitted interaction feature, the first reflected contribution feature, and the second transmitted global semantic aggregation feature, and generates a second reflected interaction feature based on the first reflected interaction feature, the first transmitted contribution feature, and the second reflected global semantic aggregation feature;

[0196] The multi-scale combined difference feature fusion module determines a second combined difference feature according to the second transmitted interaction feature and the second reflected interaction feature;

[0197] According to the inter-stream information interaction module, based on the second transmitted interaction feature, the second reflected interaction feature, and the second combined difference feature, output a second reflected contribution feature and a second transmitted contribution feature;

[0198] Based on the second transmission interaction feature and the second reflection interaction feature respectively through the bidirectional pooling feature aggregation module, the third transmission global semantic aggregation feature and the third reflection global semantic aggregation feature are output correspondingly;

[0199] Input the second transmission interaction feature, the second reflection contribution feature and the third transmission global semantic aggregation feature into the in-stream multi-scale channel interaction module to generate the third transmission interaction feature;

[0200] Input the second reflection interaction feature, the second transmission contribution feature and the third reflection global semantic aggregation feature into the in-stream multi-scale channel interaction module to determine the third reflection interaction feature;

[0201] Perform weighted fusion on the transmission flow feature, the first transmission interaction feature, the second transmission interaction feature and the second transmission interaction feature, and output the improved transmission flow feature;

[0202] Perform weighted fusion according to the reflection flow feature, the first reflection interaction feature, the second reflection interaction feature and the third reflection interaction feature to generate the improved reflection flow feature.

[0203] Furthermore, the multi-scale difference feature fusion module includes a 1×1 convolutional layer, a 3×3 convolutional layer, a 3×3 depthwise separable convolutional layer, a 5×5 depthwise separable convolutional layer and a 7×7 depthwise separable convolutional layer, a grouped convolutional layer and a sigmoid activation function; the processing process of the multi-scale difference feature fusion module includes:

[0204] After subtracting the input transmission feature and reflection feature element by element, perform convolution extraction and non-linear mapping through a cascaded 3×3 convolutional layer and a sigmoid activation function, and output the difference position feature;

[0205] Use a 1×1 convolutional layer, a 3×3 depthwise separable convolutional layer, a 5×5 depthwise separable convolutional layer and a 7×7 depthwise separable convolutional layer to parallelly extract features from the input transmission feature and then splice them, and input them into a 1×1 convolutional layer for processing to generate the transmission multi-scale feature;

[0206] Multiply the input transmission feature and the difference position feature element by element, and then add them to the transmission multi-scale feature element by element to determine the transmission difference feature;

[0207] Use a 1×1 convolutional layer, a 3×3 depthwise separable convolutional layer, a 5×5 depthwise separable convolutional layer and a 7×7 depthwise separable convolutional layer to parallelly extract features from the input reflection feature, and then splice and input them into a 1×1 convolutional layer for processing to output the reflection multi-scale feature;

[0208] Multiply the input reflection feature and the difference position feature element by element, and then add them to the reflection multi-scale feature element by element to determine the reflection difference feature;

[0209] After grouping and splicing the transmission difference features and the reflection difference features and inputting them into the grouped convolution layer for feature processing, multiply them element-wise with the difference position features and use a 3×3 convolution layer to process the output to obtain the combined difference features.

[0210] Furthermore, the in-stream multi-scale channel interaction module includes a normalization layer, a 1×1 convolution layer, a grouped convolution layer, a 3×3 depthwise separable convolution layer, a 5×5 depthwise separable convolution layer, a 7×7 depthwise separable convolution layer, a softmax activation function, a GELU activation function, and a batch normalization layer; the processing process of the in-stream multi-scale channel interaction module includes:

[0211] After normalizing the input transmission features or reflection features through the normalization layer, input them into the 1×1 convolution layer to project them into in-stream value keys;

[0212] After normalizing the input global semantic aggregation features through the normalization layer, use the 1×1 convolution layer to project them into in-stream query keys and in-stream index keys respectively, and calculate the cosine similarity using the in-stream query keys and in-stream index keys;

[0213] Based on the softmax activation function, perform attention calculation using the in-stream query keys, in-stream index keys, and in-stream value keys to determine the in-stream attention features;

[0214] After multiplying the in-stream attention features and the cosine similarity point by point, add them element-wise to the input transmission features or reflection features to generate self-adjusted features;

[0215] After channel splicing the input contribution features and the self-adjusted features, perform convolution processing through the grouped convolution layer to output the in-stream fusion features;

[0216] After parallel feature extraction of the in-stream fusion features through the 3×3 depthwise separable convolution layer, 5×5 depthwise separable convolution layer, and 7×7 depthwise separable convolution layer, input them into the batch normalization layer for processing to determine multiple in-stream scale features;

[0217] After channel splicing each in-stream scale feature and the in-stream fusion feature, use the grouped convolution layer for convolution fusion to generate the depth fusion features;

[0218] After feature enhancement of the depth fusion features through the cascaded 1×1 convolution layer, batch normalization layer, and GELU activation function, add them element-wise to the depth fusion features to output the next layer of transmission features or reflection features.

[0219] Furthermore, the inter-stream information interaction module includes a feature fusion block, a layer normalization layer, a two-dimensional convolution layer, a pooling layer, a proxy attention layer, a 1×1 depthwise separable convolution layer, and a gated mixing layer; the processing process of the inter-stream information interaction module includes:

[0220] The input transmission features and the input combined difference features, and the input reflection features and the combined difference features are respectively input into the feature fusion block in pairs for feature fusion, and the transmission difference fusion features and the reflection difference fusion features are correspondingly generated;

[0221] The transmission difference fusion features are projected into a transmission index key and a transmission value key through a two-dimensional convolutional layer, and the reflection difference fusion features are projected into a reflection index key and a reflection value key;

[0222] After performing layer normalization on the reflection difference fusion features using a layer normalization layer, they are linearly mapped into a transmission query key through a two-dimensional convolutional layer and input into a pooling layer for 1 / 4 upsampling in two dimensions to determine the transmission proxy key;

[0223] Based on performing layer normalization on the transmission difference fusion features using a layer normalization layer, they are linearly mapped into a reflection query key through a two-dimensional convolutional layer and input into a pooling layer for 1 / 4 upsampling in two dimensions to determine the reflection proxy key;

[0224] The proxy attention layer performs proxy attention calculation based on the transmission index key, the transmission value key, the transmission query key, and the transmission proxy key, and outputs the transmission attention features;

[0225] Based on the proxy attention layer, proxy attention calculation is performed using the reflection index key, the reflection value key, the reflection query key, and the reflection proxy key, and the reflection attention features are output;

[0226] After adjusting the channel semantics of the input transmission features using a 1×1 depthwise separable convolutional layer, they are input into a gated mixing layer for feature fusion with the reflection attention features to generate the reflection contribution features;

[0227] After adjusting the channel semantics of the input reflection features using a 1×1 depthwise separable convolutional layer, they are fused with the transmission attention features through a gated mixing layer to generate the transmission contribution features.

[0228] An embodiment of the present invention also provides a computer device, including a memory and a processor, and a computer program is stored in the memory; when the computer program is executed by the processor, the processor executes the steps of the single-image reflection removal method in any of the above embodiments.

[0229] An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program / instruction is stored, and when the computer program / instruction is executed by the processor, the steps of the single-image reflection removal method in any of the above embodiments are implemented.

[0230] Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein.

[0231] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical or other forms.

[0232] The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0233] In addition, each functional unit in various embodiments of the present invention can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0234] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks or optical discs that can store program codes.

[0235] As mentioned above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of various embodiments of the present invention.

Claims

1. A single-image reflection removal method, characterized in that, Including: After training and validating the initial reflection removal model using a synthetic atlas, fine-tuning the model based on a natural atlas to determine the target reflection removal model; The target reflection removal model includes a coarse separator, a feature encoder, a two-stream interaction module, and an upsampling layer; Input the image to be processed as the initial transmission image and the initial reflection image into the coarse separator for coarse separation, and correspondingly output the coarse transmission prediction image and the coarse reflection prediction image; Use the feature encoder to perform pyramid multi-scale feature enhancement on the coarse transmission prediction image and the coarse reflection prediction image respectively, and correspondingly determine the transmission stream features and the reflection stream features; Use the two-stream interaction module to perform in-stream and inter-stream feature enhancement interaction according to the transmission stream features and the reflection stream features, and correspondingly generate improved transmission stream features and improved reflection stream features; Input the improved transmission stream features and the improved reflection stream features into the upsampling layer for upsampling respectively, and correspondingly output the transmission prediction image and the reflection prediction image.

2. The single-image reflection removal method according to claim 1, characterized in that, The coarse separator includes an encoder, a bottleneck layer, and a decoder connected in sequence; The encoder includes a cascade of multiple residual modules and a max pooling layer, the decoder includes a cascade of multiple residual modules and a pixel shuffle layer, and the residual modules of the encoder and the decoder are connected by skip connections; The residual module includes a series of multiple convolutional blocks, and the convolutional block includes a 3×3 convolutional layer with residual connection, a batch normalization layer, and a ReLU activation function.

3. The single-image reflection removal method according to claim 1, wherein The feature encoder includes a SwimTransformer module, a multi-scale feature extraction module, a cross-layer fusion module, and a 1×1 convolutional layer; the processing process of the feature encoder includes: Perform feature encoding layer by layer on the feature encoding input image using a cascade of multiple Swim Transformer modules to generate encoded features in sequence; Each of the encoded features is respectively subjected to multi-scale feature enhancement through the multi-scale feature extraction module to correspondingly determine enhanced features; Based on the cross-layer fusion module, after upsampling and fusing according to each of the enhanced features, input it into the 1×1 convolutional layer for convolutional processing to output the encoded stream features.

4. The single-image reflection removal method according to claim 1, wherein The two-stream interaction module includes a bidirectional pooling feature aggregation module, a multi-scale difference feature fusion module, an in-stream multi-scale channel interaction module, and an inter-stream information interaction module; the process of using the two-stream interaction module to perform in-stream and inter-stream feature enhancement interaction according to the transmission stream features and the reflection stream features to correspondingly generate improved transmission stream features and improved reflection stream features includes: Use the bidirectional pooling feature aggregation module to perform bidirectional pooling on the transmission stream features and the reflection stream features respectively to concentrate the global semantic information, and correspondingly output the first transmission global semantic aggregation feature and the first reflection global semantic aggregation feature; Input the transmission stream features and the first transmission global semantic aggregation feature, and the reflection stream features and the first reflection global semantic aggregation feature into the in-stream multi-scale channel interaction module in pairs for feature interaction and fusion, and correspondingly generate the first transmission interaction feature and the first reflection interaction feature; Use the multi-scale combined difference feature fusion module to extract the difference information from the first transmission interaction feature and the first reflection interaction feature to determine the first combined difference feature; According to the inter-stream information interaction module, based on the proxy attention mechanism, using the first combined difference feature as a clue to guide the first transmission interaction feature and the first reflection interaction feature to perform feature interaction, and output the first reflection contribution feature and the first transmission contribution feature; Input the first transmission interaction feature and the first reflection interaction feature into the bidirectional pooling feature aggregation module respectively, and correspondingly output the second transmission global semantic aggregation feature and the second reflection global semantic aggregation feature; Through the intra-stream multi-scale channel interaction module, generate the second transmission interaction feature based on the first transmission interaction feature, the first reflection contribution feature and the second transmission global semantic aggregation feature, and generate the second reflection interaction feature based on the first reflection interaction feature, the first transmission contribution feature and the second reflection global semantic aggregation feature; Use the multi-scale combined difference feature fusion module to determine the second combined difference feature according to the second transmission interaction feature and the second reflection interaction feature; According to the inter-stream information interaction module, based on the second transmission interaction feature, the second reflection interaction feature and the second combined difference feature, output the second reflection contribution feature and the second transmission contribution feature; Through the bidirectional pooling feature aggregation module, respectively based on the second transmission interaction feature and the second reflection interaction feature, correspondingly output the third transmission global semantic aggregation feature and the third reflection global semantic aggregation feature; Input the second transmission interaction feature, the second reflection contribution feature and the third transmission global semantic aggregation feature into the intra-stream multi-scale channel interaction module to generate the third transmission interaction feature; Input the second reflection interaction feature, the second transmission contribution feature and the third reflection global semantic aggregation feature into the intra-stream multi-scale channel interaction module to determine the third reflection interaction feature; Perform weighted fusion on the transmission flow feature, the first transmission interaction feature, the second transmission interaction feature and the second transmission interaction feature, and output the improved transmission flow feature; Perform weighted fusion according to the reflection flow feature, the first reflection interaction feature, the second reflection interaction feature and the third reflection interaction feature to generate the improved reflection flow feature.

5. The single-image reflection removal method according to claim 4, wherein The multi-scale difference feature fusion module includes a 1×1 convolutional layer, a 3×3 convolutional layer, a 3×3 depthwise separable convolutional layer, a 5×5 depthwise separable convolutional layer and a 7×7 depthwise separable convolutional layer, a grouped convolutional layer and a sigmoid activation function; the processing process of the multi-scale difference feature fusion module includes: After subtracting the input transmission feature and reflection feature element by element, perform convolutional extraction and non-linear mapping through a cascaded 3×3 convolutional layer and a sigmoid activation function, and output the difference position feature; Use a 1×1 convolutional layer, a 3×3 depthwise separable convolutional layer, a 5×5 depthwise separable convolutional layer and a 7×7 depthwise separable convolutional layer to parallelly extract features from the input transmission feature and then splice them, and input them into a 1×1 convolutional layer for processing to generate the transmission multi-scale feature; Multiply the input transmission feature and the difference position feature element by element, and add them to the transmission multi-scale feature element by element to determine the transmission difference feature; After parallel feature extraction of the input reflection features through 1×1 convolutional layers, 3×3 depthwise separable convolutional layers, 5×5 depthwise separable convolutional layers, and 7×7 depthwise separable convolutional layers, the processed output reflection multi-scale features of the input 1×1 convolutional layer are concatenated; After multiplying the input reflection features element-wise with the difference location features and then adding them element-wise to the reflection multi-scale features, the reflection difference features are determined; The transmission difference features and the reflection difference features are grouped and concatenated, and after being input into a grouped convolutional layer for feature processing, they are multiplied element-wise with the difference location features and processed by a 3×3 convolutional layer to output combined difference features.

6. The single-image reflection removal method according to claim 4, wherein The intra-stream multi-scale channel interaction module includes a normalization layer, 1×1 convolutional layers, grouped convolutional layers, 3×3 depthwise separable convolutional layers, 5×5 depthwise separable convolutional layers, and 7×7 depthwise separable convolutional layers, softmax activation functions, GELU activation functions, and batch normalization layers; the processing process of the intra-stream multi-scale channel interaction module includes: After normalizing the input transmission features or reflection features through the normalization layer, they are input into a 1×1 convolutional layer to be projected into intra-stream value keys; After normalizing the input global semantic aggregation features through the normalization layer, they are projected into intra-stream query keys and intra-stream index keys respectively by a 1×1 convolutional layer, and the cosine similarity is calculated using the intra-stream query keys and the intra-stream index keys; Based on the softmax activation function, attention calculation is performed using the intra-stream query keys, the intra-stream index keys, and the intra-stream value keys to determine intra-stream attention features; After multiplying the intra-stream attention features with the cosine similarity and then adding them element-wise to the input transmission features or reflection features, self-adjusted features are generated; After channel concatenating the input contribution features with the self-adjusted features and then performing convolutional processing through a grouped convolutional layer, intra-stream fusion features are output; After parallel feature extraction of the intra-stream fusion features through 3×3 depthwise separable convolutional layers, 5×5 depthwise separable convolutional layers, and 7×7 depthwise separable convolutional layers, they are input into a batch normalization layer for processing to determine multiple intra-stream scale features; After channel concatenating each of the intra-stream scale features with the intra-stream fusion features and then performing convolutional fusion using a grouped convolutional layer, depth fusion features are generated; After enhancing the features of the depth fusion features through cascaded 1×1 convolutional layers, batch normalization layers, and GELU activation functions and then adding them element-wise to the depth fusion features, the transmission features or reflection features of the next layer are output.

7. The single-image reflection removal method according to claim 4, wherein The inter-stream information interaction module includes a feature fusion block, layer normalization layers, two-dimensional convolutional layers, pooling layers, proxy attention layers, 1×1 depthwise separable convolutional layers, and gated mixing layers; The processing process of the inter-stream information interaction module includes: The input transmission features and the input combined difference features, and the input reflection features and the combined difference features are respectively input in pairs into the feature fusion block for feature fusion, and transmission difference fusion features and reflection difference fusion features are correspondingly generated; Project the transmitted difference fusion feature into a transmitted index key and a transmitted value key through a two-dimensional convolutional layer, and project the reflected difference fusion feature into a reflected index key and a reflected value key; After performing layer normalization on the reflected difference fusion feature using a layer normalization layer, linearly map it into a transmitted query key through a two-dimensional convolutional layer and input it into a pooling layer for 1 / 4 upsampling in two dimensions to determine a transmitted proxy key; Based on performing layer normalization on the transmitted difference fusion feature using a layer normalization layer, linearly map it into a reflected query key through a two-dimensional convolutional layer and input it into a pooling layer for 1 / 4 upsampling in two dimensions to determine a reflected proxy key; Perform proxy attention calculation through a proxy attention layer according to the transmitted index key, the transmitted value key, the transmitted query key, and the transmitted proxy key, and output a transmitted attention feature; Based on the proxy attention layer, perform proxy attention calculation using the reflected index key, the reflected value key, the reflected query key, and the reflected proxy key, and output a reflected attention feature; After adjusting the channel semantics of the input transmitted feature using a 1×1 depthwise separable convolutional layer, input it into a gated mixing layer to perform feature fusion with the reflected attention feature to generate a reflected contribution feature; After adjusting the channel semantics of the input reflected feature using a 1×1 depthwise separable convolutional layer, perform feature fusion with the transmitted attention feature through a gated mixing layer to generate a transmitted contribution feature.

8. A single-image reflection removal device, characterized in that, Comprises: A model training module for validating the model training of the initial reflection removal model using a synthetic atlas set, and then performing model fine-tuning based on a natural atlas set to determine a target reflection removal model; The target reflection removal model includes a coarse separator, a feature encoder, a two-stream interaction unit, and an upsampling layer; An image coarse separation module for respectively inputting a to-be-processed image as an initial transmitted image and an initial reflected image into the coarse separator for coarse separation, and correspondingly outputting a transmitted coarse prediction image and a reflected coarse prediction image; A feature encoding module for respectively performing pyramid multi-scale feature enhancement on the transmitted coarse prediction image and the reflected coarse prediction image through the feature encoder to correspondingly determine a transmitted stream feature and a reflected stream feature; A feature interaction module for performing in-stream and inter-stream feature enhancement interaction on the transmitted stream feature and the reflected stream feature using the two-stream interaction unit to correspondingly generate an improved transmitted stream feature and an improved reflected stream feature; A prediction output module for respectively inputting the improved transmitted stream feature and the improved reflected stream feature into the upsampling layer for upsampling to correspondingly output a transmitted prediction image and a reflected prediction image.

9. A computer device, characterized in that, Comprises a memory and a processor. When a computer program stored in the memory is executed by the processor, the processor executes the steps of the single-image reflection removal method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, the steps of the single-image reflection removal method according to any one of claims 1-7 are implemented.

Images