Image processing method, apparatus, and electronic device

Abstract

Embodiments of the present application disclose an image processing method and device and electronic equipment. In the method, frequency domain information and spatial information of image features output by a first down-sampling layer are used to obtain global frequency domain features, and image features and windows of multiple scales output by a second down-sampling layer are used to obtain multi-scale features, so that in the process of up-sampling to obtain an output image, the obtained global frequency domain features and multi-scale features can be combined, and a neural network model can more accurately and comprehensively determine reflection content to be removed in an input image, so that the output image has a better reflection content removal effect.

Description

[0001] Cross-references to related applications This application claims priority to U.S. Patent Application No. 63 / 754,493, entitled “U-SHAPED FFT TRANSFORMER AND HIERARCHICAL TRANSFORMER FOR REFLECTION REMOVAL,” filed February 5, 2025, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer technology, and more specifically, to an image processing method, apparatus, and electronic device. Background Technology

[0003] When capturing images or videos through a mirror (e.g., a glass window), the images often contain mirrored objects, severely impacting image quality. With the widespread application of deep learning methods in image processing, neural networks can be used to remove reflective content from images, resulting in reflection-free images. However, the effectiveness of these reflection removal methods still needs improvement. Summary of the Invention

[0004] In view of the above problems, this application proposes an image processing method, apparatus, and electronic device to improve the above problems.

[0005] In a first aspect, this application provides an image processing method applied to a neural network model, the neural network model including multiple downsampling layers and multiple upsampling layers. The method includes: downsampling an input image through the multiple downsampling layers to obtain a first image feature, wherein the first downsampling layer processes the input image, and each subsequent downsampling layer processes the output of the adjacent preceding downsampling layer; obtaining a global frequency domain feature using the frequency domain information and spatial information of the image feature output by the first downsampling layer, wherein the first downsampling layer is one of the multiple downsampling layers; obtaining a multi-scale feature using the image feature output by the second downsampling layer and windows of multiple scales, wherein the second downsampling layer is one of the multiple downsampling layers; and upsampling the first image feature using the multiple upsampling layers, the global frequency domain feature, and the multi-scale feature to obtain an output image with reflection content removed.

[0006] Secondly, this application provides an image processing apparatus for running a neural network model, the neural network model including multiple downsampling layers and multiple upsampling layers. The method includes: a downsampling unit for downsampling an input image through the multiple downsampling layers to obtain a first image feature, wherein the first downsampling layer processes the input image, and each subsequent downsampling layer processes the output of the adjacent preceding downsampling layer; a global frequency feature acquisition unit for obtaining global frequency domain features through the frequency domain information and spatial information of the image features output by the first downsampling layer, wherein the first downsampling layer is one of the multiple downsampling layers; and a multi-scale feature acquisition unit for obtaining multi-scale features through the image features output by the second downsampling layer and windows of multiple scales, wherein the second downsampling layer is one of the multiple downsampling layers. The upsampling unit is used to upsample the first image features through the multiple upsampling layers, the global frequency domain features, and the multi-scale features to obtain an output image with reflection content removed.

[0007] Thirdly, this application provides an electronic device, which includes at least a processor and a memory; one or more programs are stored in the memory and configured to be executed by the processor to implement the above-described method.

[0008] Fourthly, this application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the above-described method.

[0009] Fifthly, this application provides a computer-readable storage medium storing program code, wherein the above-described method is executed when the program code is run by a processor.

[0010] This application provides an image processing method, apparatus, and electronic device. In this method, an input image is first downsampled through multiple downsampling layers to obtain a first image feature. A global frequency domain feature can be obtained using the frequency domain and spatial information of the image features output by the first downsampling layer, where the first downsampling layer is one of the multiple downsampling layers. A multi-scale feature is obtained using the image features output by a second downsampling layer and windows of multiple scales, where the second downsampling layer is one of the multiple downsampling layers. Then, the first image feature is upsampled using the multiple upsampling layers, the global frequency domain feature, and the multi-scale feature to obtain an output image with reflections removed. Thus, by means of the above method, when downsampling the input image, global frequency domain features are obtained through the frequency domain information and spatial information of the image features output by the first downsampling layer, and multi-scale features are obtained through the image features output by the second downsampling layer and windows of multiple scales. Therefore, during the upsampling process to obtain the output image, the obtained global frequency domain features and multi-scale features can be combined to enable the neural network model to more accurately and comprehensively determine the reflection content that needs to be removed from the input image, so that the output image has a better reflection content removal effect. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A schematic diagram of an application environment for the image processing method proposed in an embodiment of this application is shown; Figure 2 A schematic diagram illustrating another application environment of the image processing method proposed in the embodiments of this application is shown; Figure 3 A flowchart of an image processing method proposed in an embodiment of this application is shown; Figure 4 This paper shows a schematic diagram of the structure of the first feature extraction module in this application; Figure 5 This paper shows a schematic diagram of one structure of the downsampling layer in this application; Figure 6 A schematic diagram illustrating the removal of reflected content in this application is shown; Figure 7 A flowchart of an image processing method according to another embodiment of this application is shown; Figure 8This paper shows a schematic diagram of one structure of the Fourier feature extraction layer in this application; Figure 9 A schematic diagram of one structure of the first feedforward layer (multi-scale convolutional feedforward network) in this application is shown; Figure 10 This paper shows a schematic diagram of the structure of the second feature extraction module in this application; Figure 11 A flowchart of an image processing method according to another embodiment of this application is shown; Figure 12 This paper shows a schematic diagram of a layered window self-attention layer structure in this application; Figure 13 This paper shows a schematic diagram of the structure of the third feature extraction module in this application; Figure 14 A schematic diagram of one structure of the second feedforward layer in this application is shown; Figure 15 This invention provides a schematic diagram of the structure of a neural network model. Figure 16 A structural block diagram of an image processing apparatus according to another embodiment of this application is shown; Figure 17 This invention illustrates a structural block diagram of an electronic device for performing an image processing method according to an embodiment of the present invention; Figure 18 This is a storage unit in this application embodiment for storing or carrying program code that implements the image processing method according to this application embodiment. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0014] Single Image Reflection Removal (SIRR) technology plays a crucial role in image processing, eliminating unwanted reflections in the background. These reflections can occur when images are captured through a glass surface and can severely degrade image quality. The inventors of this application have found that removing reflections from images remains a challenging problem due to the complexity and diversity of reflections encountered in real-world scenes. For example, the intensity, shape, light source, size, and coverage of these reflections vary significantly in images, making it difficult for existing methods to effectively handle all situations.

[0015] Therefore, after discovering the above-mentioned problems in their research, the inventors proposed the image processing method, apparatus, and electronic device described in this application, which can improve upon these problems. In this method, the input image is first downsampled through multiple downsampling layers to obtain a first image feature. Furthermore, a global frequency domain feature can be obtained using the frequency domain and spatial information of the image features output by the first downsampling layer, where the first downsampling layer is one of the multiple downsampling layers. Then, a multi-scale feature is obtained using the image features output by a second downsampling layer and windows of multiple scales, where the second downsampling layer is one of the multiple downsampling layers. Finally, the first image feature is upsampled using the multiple upsampling layers, the global frequency domain feature, and the multi-scale feature to obtain an output image with reflections removed. Thus, by means of the above method, when downsampling the input image, global frequency domain features are obtained through the frequency domain information and spatial information of the image features output by the first downsampling layer, and multi-scale features are obtained through the image features output by the second downsampling layer and windows of multiple scales. Therefore, during the upsampling process to obtain the output image, the obtained global frequency domain features and multi-scale features can be combined to enable the neural network model to more accurately and comprehensively determine the reflection content that needs to be removed from the input image, so that the output image has a better reflection content removal effect.

[0016] Before providing a more detailed description of the embodiments of this application, an application environment related to the embodiments of this application will be introduced.

[0017] The application environment involved in the embodiments of this application will be introduced below.

[0018] In the embodiments of this application, the provided image processing method can be executed by an electronic device. In this manner, all steps of the image processing method provided in the embodiments of this application can be performed by the electronic device. For example, as Figure 1 As shown, in the image processing method provided in this application embodiment, where all steps can be executed by an electronic device, all steps can be executed by an application program in the electronic device 100. This application program can be an image processing program or an image acquisition program.

[0019] Furthermore, the image processing method provided in this application embodiment can also be executed by a server. In this server-executed manner, the server can begin executing the steps of the image processing method provided in this application embodiment in response to a triggering instruction. This triggering instruction can be sent by an electronic device used by a user, or it can be triggered locally by the server in response to some automated event.

[0020] Furthermore, the image processing method provided in this application embodiment can also be executed collaboratively by an electronic device and a server. In this method, some steps of the image processing method provided in this application embodiment are executed by the electronic device, while other steps are executed by the server. For example, taking the image processing method in this application as an example... Figure 2 As shown, the electronic device 100 can perform an image processing method including: downsampling an input image through a plurality of downsampling layers to obtain a first image feature. In the plurality of downsampling layers, the first downsampling layer processes the input image, and each subsequent downsampling layer processes the output of the adjacent preceding downsampling layer. Then, the electronic device 100 can transmit the first image feature to the server 200, which then performs the remaining steps to obtain an output image. Afterwards, the server 200 can either save the output image locally or return it to the electronic device 100 for storage or display.

[0021] It should be noted that in this method where electronic devices and servers work together, the steps performed by the electronic devices and servers are not limited to those described in the examples above. In practical applications, the steps performed by the electronic devices and servers can be dynamically adjusted according to the actual situation.

[0022] It should be noted that the electronic device 100 is not specifically limited in this embodiment. For example, the electronic device can be a smartphone, tablet, wearable device, vehicle terminal, etc. The wearable device can be smart glasses or AR (Augmented Reality) glasses. The server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. In the case where the image processing method provided in this embodiment is executed by a server cluster or distributed system composed of multiple physical servers, different steps in the image processing method can be executed by different physical servers, or can be executed in a distributed manner by servers built on a distributed system.

[0023] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0024] Please see Figure 3 This application provides an image processing method applied to a neural network model, which includes multiple downsampling layers and multiple upsampling layers. The method includes: S110: The input image is downsampled through multiple downsampling layers to obtain the first image features. In the multiple downsampling layers, the first downsampling layer processes the input image, and each subsequent downsampling layer processes the output of the adjacent preceding downsampling layer.

[0025] In this embodiment, the input image can be understood as the image to be processed for reflection removal. The reflection content can be understood as the image components that interfere with the target background information when an image is captured through a transparent or semi-transparent medium such as glass. In one case, the input image may include a reflective layer (R) and a transmissive layer (T), wherein the reflective layer contains the reflection content to be removed. The solution provided in this application can make the generated transmissive layer as close as possible to the corresponding true value, i.e., an ideal reflection-free image. Optionally, the input image can be obtained by image acquisition using an image acquisition device (e.g., a camera).

[0026] In this embodiment, the downsampling layer is a core operation in the neural network model that optimizes model performance by reducing the dimension (length and width) of the feature map space. It can retain key information while reducing computational cost and improving feature abstraction.

[0027] In one approach, each downsampling layer includes at least one first feature extraction module. Optionally, the first feature extraction module can be a nonlinear activation-free (NAF) block.

[0028] Optionally, the first feature extraction module may include a first normalization layer, a convolutional layer, and a simplified channel attention layer. The first normalization layer may be a layer normalization layer. This first normalization layer normalizes the input (e.g., the input image or features output from the previous first feature extraction module) along the channel dimension, adjusting the feature values ​​to a similar distribution range and avoiding gradient vanishing or training instability caused by feature value fluctuations during training.

[0029] Convolutional layers can extract local features from the input.

[0030] The Simplified Channel Attention (SCA) layer assigns channel-level weights to local features extracted by the convolutional layers. By calculating the importance score of each channel, it strengthens key feature channels (such as channels corresponding to reflective region edges and non-reflective background textures) and suppresses redundant channels. Optionally, the SCA layer can act as a feature filter to address the issue of convolution treating all channels equally, allowing the model to focus more on features valuable to the task (such as reflection removal and image denoising involved in the embodiments of this application), thereby improving the specificity of feature representation.

[0031] In this approach, the processing steps of the first feature extraction module include: processing the image features input to the first feature extraction module through a first normalization layer, and transmitting the image features output by the first normalization layer to a convolutional layer; after processing the input image features in the convolutional layer, transmitting the image features output by the convolutional layer to a simplified channel attention layer; after processing the input image features in the simplified channel attention layer, obtaining the output of the first feature extraction module through the output of the simplified channel attention layer and the image features input to the first feature extraction module.

[0032] The first feature extraction module integrates layer normalization for stable training, employs deep convolution to improve efficiency, highlights key features through a simplified channel attention mechanism, and dynamically adjusts features using gated linear units. By removing non-linear activation functions, the first feature extraction module further simplifies the model structure while maintaining strong feature representation capabilities.

[0033] For example, please refer to Figure 4 , Figure 4 An exemplary structure of the first feature extraction module is shown. Figure 4 The first feature extraction module in the structure shown includes a first part and a second part.

[0034] The first part, in the order of processing, may include a first normalization layer, a convolutional layer (e.g., a 1×1 conv), a deconvolutional layer (e.g., a 3×3 dconv), a simple gate layer, a simplified channel attention layer, and a convolutional layer. The gate layer may be a gated linear unit (GLU) layer. The second part, in the order of processing, may include a first normalization layer, a convolutional layer (e.g., a 1×1 conv), a simple gate layer, and a convolutional layer (e.g., a 1×1 conv).

[0035] Among them, such as Figure 4As shown, in the first feature extraction module, the input of the first part can be connected to the output residual of the first part. That is, the input of the first part and the output residual of the first part can be added together and used as the input of the second part. Correspondingly, in the first feature extraction module, the input of the second part can be connected to the output residual of the second part. That is, the input of the second part and the output residual of the second part can be added together and used as the output of the first feature extraction module.

[0036] Optionally, the number of first feature extraction modules included in each downsampling layer can be one or more. Furthermore, the number of first feature extraction modules included in each downsampling layer can be different. Where a downsampling layer includes multiple first feature extraction modules, the output of the preceding first feature extraction module serves as the input of the adjacent following first feature extraction module. "Adjacent following" can be understood as being adjacent in sorting order and relatively following. For example, as shown... Figure 5 As shown, in Figure 5 The downsampling layer C1 shown includes a first processing module Y1, a first processing module Y2, and a first processing module Y3. The first processing module Y2 is the first processing module adjacent to the first processing module Y1, and the first processing module Y3 is the first processing module adjacent to the first processing module Y2.

[0037] S120: Global frequency domain features are obtained by using the frequency domain information and spatial information of the image features output by the first downsampling layer. The first downsampling layer is one of multiple downsampling layers.

[0038] In this embodiment, the first downsampling layer can be any one of multiple downsampling layers. For example, the first downsampling layer can be the second, third, or last downsampling layer. The image features output by the first downsampling layer can be further analyzed using its frequency domain information and spatial information to obtain corresponding global frequency domain features. Spatial information can capture local features, while frequency domain information can characterize more global features. By fusing the features corresponding to these two types of information, this application can more effectively capture both local and global dependencies simultaneously, thereby improving the performance of the reflection removal task.

[0039] S130: Multi-scale features are obtained by using the image features output by the second downsampling layer and windows of multiple scales. The second downsampling layer is one of multiple downsampling layers.

[0040] In this embodiment, the second downsampling layer can be any one of multiple downsampling layers. For example, the first downsampling layer can be the second, third, or last downsampling layer. The second and first downsampling layers can be the same layer or different layers. Optionally, in the case of four downsampling layers, the first downsampling layer can be the second layer, and the second layer can be the last layer. By using windows of multiple scales, local and global information can be effectively considered, enabling cross-scale feature aggregation and establishing long-range dependencies.

[0041] S130 and S120 can be executed simultaneously or sequentially.

[0042] S140: The first image features are upsampled through multiple upsampling layers, global frequency domain features, and multi-scale features to obtain an output image with reflection content removed.

[0043] It should be noted that the neural network model mentioned in the embodiments of this application can be a model that has already been trained with training data. Therefore, the neural network model already has the ability to identify reflective content in an image. Optionally, the training data may include multiple image pairs, wherein the content of the two images in each image pair is the same or similar, except that one image includes reflective content while the other image does not.

[0044] After the neural network model is trained on the aforementioned training data, its multiple upsampling layers also acquire the ability to identify reflections. Therefore, reflections can be removed progressively during the upsampling process, resulting in an output image that is the input image with reflections removed. Specifically, the obtained global frequency domain features and multi-scale features can more clearly identify the characteristics of reflections in the input image, enabling more comprehensive and accurate removal of reflections during upsampling, thus improving the effectiveness of reflection removal.

[0045] It should be noted that when there are multiple upsampling layers, there are no specific limitations on the application stages of global frequency domain features and multi-scale features. In one approach, multi-scale features can be applied to the first upsampling layer among multiple upsampling layers. That is, during the upsampling process of the first upsampling layer, the first image features are upsampled based on multi-scale features. In another approach, global frequency domain features can be applied to an intermediate upsampling layer (the upsampling layer between the first and last upsampling layers). That is, this intermediate upsampling layer can process the features output by the previous upsampling layer based on global frequency domain features and transmit the processing result to the next upsampling layer.

[0046] Please see Figure 6 This is a schematic diagram illustrating the effect of removing reflective content by executing the method provided in the embodiments of this application. Figure 6 In the image shown, there is reflected content in the area enclosed by the wireframe in the left image. Figure 6 The image on the right shows the area in the image on the left where there was reflection, and how it looks after the reflection has been removed.

[0047] This embodiment provides an image processing method that, through the above-described manner, when downsampling the input image, obtains global frequency domain features by using the frequency domain information and spatial information of the image features output by the first downsampling layer, and obtains multi-scale features by using the image features output by the second downsampling layer and windows of multiple scales. Thus, during the upsampling process to obtain the output image, the obtained global frequency domain features and multi-scale features can be combined to enable the neural network model to more accurately and comprehensively determine the reflection content that needs to be removed from the input image, so that the output image has a better reflection content removal effect.

[0048] Please see Figure 7 This application provides an image processing method applied to a neural network model. The neural network model includes multiple downsampling layers, multiple upsampling layers, and a second feature extraction module. The method includes: S210: The input image is downsampled through multiple downsampling layers to obtain the first image features. In the multiple downsampling layers, the first downsampling layer processes the input image, and each subsequent downsampling layer processes the output of the adjacent preceding downsampling layer.

[0049] S220: The second feature extraction module obtains the frequency domain features and spatial features of the image features output by the first downsampling layer. The frequency domain features represent the frequency domain information of the image features output by the first downsampling layer, and the spatial features represent the spatial information of the image features output by the first downsampling layer.

[0050] In this implementation, the second feature extraction module can be understood as a module obtained by introducing the Fast Fourier Transform (FFT) into the Transformer. In this case, the second feature extraction module can be called the F2T2 (Fast Fourier Transform Transformer) block. This design implements a dual-domain hybrid structure, which can be used for multi-scale receptive field modeling. The spatial domain handles local feature extraction, while the frequency domain focuses on global modeling. By combining these two domains, local and global dependencies are effectively captured, improving the performance of the reflection removal task.

[0051] S230: The frequency domain features and spatial features are fused through the second feature extraction module to obtain global frequency domain features.

[0052] S240: Multi-scale features are obtained by using the image features output by the second downsampling layer and windows of multiple scales. The second downsampling layer is one of multiple downsampling layers.

[0053] S250: The first image features are upsampled through multiple upsampling layers, global frequency domain features, and multi-scale features to obtain an output image with reflection content removed.

[0054] In one approach, the second feature extraction module includes a Fourier feature extraction layer (FFT layer), which comprises a spatial feature extraction module, a frequency domain feature extraction module, and a fusion module. The second feature extraction module acquires frequency domain features and spatial features from the image features output by the first downsampling layer, including: acquiring spatial features from the image features output by the first downsampling layer through the spatial feature extraction module, and acquiring frequency domain features from the image features output by the first downsampling layer through the frequency domain feature extraction module; the second feature extraction module then fuses the frequency domain features and spatial features to obtain global frequency domain features, including: fusing the frequency domain features and spatial features through the fusion module to obtain global frequency domain features.

[0055] For example, Figure 8 The example demonstrates the structure of a Fourier feature extraction layer. This layer includes a convolution module, a splitting module, and a spatial-domain feature extraction module. The module consists of a frequency-domain branch, a frequency-domain feature extraction module, and a fusion module based on channel attention.

[0056] The Fourier feature extraction layer includes convolutional modules that can be 1×1 convolutions. These convolutional modules can adjust the channel dimension, compressing / expanding the number of channels of the input features (e.g., features input to the Fourier feature extraction layer) to fit the dimension of subsequent convolutions.

[0057] The segmentation module is used to split the features processed by the convolution module (e.g., 1x1, conv) into two parts according to channels or dimensions. One part is sent to the spatial feature extraction module, and the other part is sent to the frequency domain feature extraction module. This allows the two modules to process the spatial information and the frequency domain information respectively. Finally, the two types of information are fused by a subsequent module (e.g., the fusion module).

[0058] In this process, the features input to the spatial feature extraction module can be first split (e.g., split based on channel dimension) into two parts, where "S" in the diagram represents feature splitting. The two split parts are then used to extract features using different convolutions, and finally fused by concatenation. The features extracted using different convolutions can carry spatial information at two scales, where "C" in the diagram represents feature concatenation. For example... Figure 8 Regarding the spatial feature extraction module, for one segment obtained from the splitting, a 3×3 convolution can be used to capture local spatial features (such as near-range dependencies of edges and textures). For the other segment obtained from the splitting, a 3×3 convolution with a dilation rate of 2 can be used. With a dilation rate of 2, the receptive field of the convolution kernel can be expanded from 3×3 to 5×5 to better capture long-range spatial features (such as far-range dependencies of large-scale structures). This dilation rate is only an example and can be dynamically adjusted according to actual conditions.

[0059] Specifically, the features input to the frequency domain feature extraction module can first be processed using a two-dimensional fast Fourier transform (2DFFT) to convert the spatial domain features (features input to the frequency domain feature extraction module) into a frequency domain complex representation. For example, it can be decomposed into two components: Real (real part) and Imaginary (imaginary part), thus realizing the conversion of spatial information to frequency domain information. The real and imaginary parts of the features can then be concatenated along the channel dimension to form a frequency domain complex feature, providing input for subsequent frequency domain processing.

[0060] The frequency domain feature extraction module also involves two operations: Frequency-conditional Position Encoding (FREPE) and Frequency Dynamic Convolution (FREV). FREPE includes 3×3 Depthwise Separable Convolution (DW Conv) and an addition operation. Its purpose is to inject position awareness into the input features (the aforementioned complex frequency features). For example, the 3×3 DW Conv can extract the local correlations of the complex frequency features, generating a position code that adapts to the current frequency domain distribution. This code is then fused with the input complex frequency features through addition, allowing the model to perceive the correlation between position and frequency in the frequency domain as well.

[0061] The features output after the frequency domain conditional location coding operation will then undergo a frequency domain dynamic convolution operation, which involves multi-branch convolution (Conv_1~Conv_n), 1×1 DW Conv, and Softmax.

[0062] The role of frequency-domain dynamic convolution is to adaptively extract multi-scale frequency-domain patterns. For example, multi-branch convolution can capture global patterns at different frequencies in parallel. The 1×1 DW Conv extracts branch decision features, which are then mapped to normalized branch weights by Softmax. Finally, the multi-branch outputs are weighted and fused, making the model's extraction of frequency-domain patterns more targeted. The output of the frequency-domain dynamic convolution operation is then processed by a two-dimensional inverse fast Fourier transform (2DIFFT) to transform it back into the spatial domain, allowing the frequency-domain information to be subsequently fused with the results of the spatial domain branches.

[0063] The features output by both the frequency domain feature extraction module and the spatial feature extraction module are processed by a Gaussian Error Linear Unit (GELU), and then fused by a channel attention-based fusion module. The fused features can then be processed by a convolutional module (e.g., 1x1, conv) to ensure that the dimension of the output features of the Fourier feature extraction layer is consistent with the dimension of the features input to the Fourier feature extraction layer.

[0064] In one approach, the second feature extraction module may also include a second normalization layer and a first feedforward layer.

[0065] Optionally, the second normalization layer can be a layer normalization layer. Optionally, the first feedforward layer can be a ConvFFN (Feed-Forward Neural Network). For example, the first feedforward layer can be a multi-kernel ConvFFN.

[0066] For example, Figure 9 The diagram illustrates the structure of a multi-scale convolutional feedforward network. The multi-scale convolutional feedforward network includes: The 1×1 convolution is used to adjust the channels of the input features, preparing for subsequent multi-branch processing. The split module (split) divides the features into three branches, each processed by a depthwise separable convolution with a different kernel size. 1×1 depthwise convolution (DW Conv): captures the finest-grained local features (small receptive field, focusing on pixel-level details); 3×3 depthwise convolution: captures medium-scale local features (moderate receptive field, focusing on small region structures); 5×5 depthwise convolution: captures large-scale local features (large receptive field, focusing on large-scale structures); the concatenation module (Concat) concatenates the outputs of the three branches along the channel dimension, fusing "multi-scale features"; the Gaussian error linear unit (GeLU) introduces non-linear expressive power into the concatenated features; and the 1×1 convolution performs channel integration on the fused features, outputting the final result.

[0067] In this approach, the image features output by the first downsampling layer can be processed by the second normalization layer to obtain the second image features; Correspondingly, obtaining spatial features from the image features output by the first downsampling layer through the spatial feature extraction module, and obtaining frequency domain features from the image features output by the first downsampling layer through the frequency domain feature extraction module, can include: obtaining spatial features of the second image features through the spatial feature extraction module, and obtaining frequency domain features of the second image features through the frequency domain feature extraction module; fusing the frequency domain features and spatial features through the fusion module to obtain global frequency domain features can include: fusing the frequency domain features and spatial features through the fusion module to obtain a first fused feature; obtaining global frequency domain features through the image features output by the first feedforward layer and the first downsampling layer, and the first fused feature.

[0068] The process of obtaining global frequency domain features through the image features output by the first feedforward layer and the first downsampling layer, and the first fusion feature, can include obtaining a second fusion feature through the image features output by the first downsampling layer and the first fusion feature. This can be achieved by performing a residual concatenation between the image features output by the first downsampling layer and the first fusion feature to obtain the second fusion feature. The second fusion feature is then combined with the first feedforward layer to obtain the first fusion feature.

[0069] For example, one structure of the second feature extraction module can be as follows: Figure 10 As shown, Figure 10 The second feature extraction module shown may include two second normalization layers, a Fourier feature extraction layer, and a first feedforward layer. The features input to the second feature extraction module are processed sequentially by the first second normalization layer and the Fourier feature extraction layer. The output of the Fourier feature extraction layer is then added to the input of the second feature extraction module, serving as the input for a subsequent part (which can be understood as the aforementioned second fusion feature). This subsequent part can be understood as the combination of the second second normalization layer and the first feedforward layer. The output of this subsequent part can be understood as the aforementioned first fusion feature.

[0070] This embodiment provides an image processing method that enables a neural network model to more accurately and comprehensively determine the reflection content to be removed from the input image, resulting in a better reflection content removal effect in the output image. Furthermore, this embodiment employs Fast Fourier Convolution (FFT), which provides a receptive field covering the entire image, thereby achieving comprehensive feature extraction. A Fourier feature extraction layer is introduced in the second feature extraction module, enabling the extraction of reflection features from both the frequency and spatial dimensions to obtain global frequency domain features. Specifically, the second feature extraction module integrates the FFT mechanism into the Transformer architecture, constructing a dual-domain hybrid structure to achieve multi-scale receptive field modeling. The spatial domain is responsible for capturing local features, while the frequency domain excels at handling global modeling. By fusing these two domains, the method provided in this embodiment can more effectively capture both local and global dependencies simultaneously, thereby improving the performance of the reflection removal task.

[0071] Please see Figure 11This application provides an image processing method applied to a neural network model. The neural network model includes multiple downsampling layers, multiple upsampling layers, and a third feature extraction module. The third feature extraction module includes a hierarchical window-based self-attention (W-SA) layer, which comprises a dual feature extraction (DEF) layer, a spatial self-correlation (S-SC) layer, and a channel self-correlation (C-SC) layer. In this embodiment, the third feature extraction module can be understood as a Hierarchical Transformer (HiT) module. The method involved in this embodiment includes: S310: The input image is downsampled through multiple downsampling layers to obtain the first image features. In the multiple downsampling layers, the first downsampling layer processes the input image, and each subsequent downsampling layer processes the output of the adjacent preceding downsampling layer.

[0072] S320: Global frequency domain features are obtained by using the frequency domain information and spatial information of the image features output by the first downsampling layer. The first downsampling layer is one of multiple downsampling layers.

[0073] S330: The image features output by the second downsampling layer are processed by the dual-domain feature extraction layer to obtain dual-domain features.

[0074] The dual-domain feature extraction layer is used to enhance the discriminative power of reflection features in both domains (spatial and channel) (the feature response of the reflection area increases while that of the background area decreases), so that the reflection content can be better distinguished from other content in subsequent steps through the dual-domain features.

[0075] S340: The dual-domain features are processed through windows of multiple scales, spatial autocorrelation layers, and channel autocorrelation layers to obtain multi-scale features.

[0076] In this embodiment, the number and specific size of windows at various scales are not specifically limited. One approach may include three window scales: 4×4, 8×8, and 16×16 pixels. When processing dual-domain features using windows of each scale, the number of windows used corresponds to the window scale (size). For example, taking a dual-domain feature with a size of H×W and a 4×4 window as an example, the number of multiple 4×4 windows used in the processing is (H / 4)×(W / 4). These multiple 4×4 windows can be tiled non-overlappingly on the dual-domain feature for simultaneous feature extraction. Correspondingly, if the grid is divided into "8×8" windows, (H / 8)×(W / 8) medium windows are obtained, and the area of ​​each medium window is 4 times that of a 4×4 window; if the grid is divided into "16×16" windows, (H / 16)×(W / 16) windows are obtained, and the area of ​​each large window is 4 times that of an 8×8 window and 16 times that of a 4×4 window.

[0077] S350: The first image features are upsampled through multiple upsampling layers, global frequency domain features, and multi-scale features to obtain an output image with reflection content removed.

[0078] One approach, which involves processing dual-domain features through windows of multiple scales, spatial autocorrelation layers, and channel autocorrelation layers to obtain multi-scale features, may include: processing the dual-domain features separately through windows of multiple scales to obtain window features for each window at each scale; then processing each window feature through spatial autocorrelation layers and channel autocorrelation layers to obtain enhanced window features for each window at each scale; concatenating the enhanced window features corresponding to each window at multiple scales to obtain multi-scale enhanced window features; and finally, obtaining the multi-scale features using the multi-scale enhanced window features and the image features output from the second downsampling layer.

[0079] For example, the layered window self-attention layer structure involved in this embodiment can be as follows: Figure 12 As shown. In Figure 12 In this process, the features input to the hierarchical window self-attention layer first pass through a dual-domain feature extraction layer, and then through a spatial autocorrelation layer and a channel autocorrelation layer. Figure 12 The feature map input to the self-attention layer of the hierarchical window has a size of C×H×W. Here, C (Channels) represents the number of channels; H (Height) represents the height of the feature map (number of pixels in the vertical direction); and W (Width) represents the width of the feature map (number of pixels in the horizontal direction).

[0080] like Figure 12 As shown, the features output by the dual-domain feature extraction layer can be split along the channel dimension and used as Q and V respectively, then input into the spatial autocorrelation layer and the channel autocorrelation layer for processing. The specific processing steps for the spatial autocorrelation layer and the channel autocorrelation layer can be described as follows: Figure 12 As shown.

[0081] Specifically, for the spatial autocorrelation layer, the spatial correlation is calculated using the inputs Q and V, along with matrix multiplication (Q×V^T), and a spatial correlation matrix is ​​output to capture the dependencies between different spatial locations. For the channel autocorrelation layer, the channel correlation is calculated using the inputs Q and V, along with matrix multiplication (Q^T×V), to capture the feature dependencies between different channels; the output channel correlation matrix enhances the information interaction between channels.

[0082] The outputs of the spatial autocorrelation layer and the channel autocorrelation layer are concatenated. Figure 12 The "C" in the diagram represents the concatenation process. After concatenation, the data is processed through a linear layer. The output of the linear layer is then reshaped before being output. Figure 12 The "R" in the equation represents reshaping; finally, through residual connections, the reshaping output features are fused with the original input (the input of the hierarchical window self-attention layer) to obtain the multi-scale output features.

[0083] Optionally, the third feature extraction module further includes a third normalization layer and a second feedforward layer. For example, one structure of the third feature extraction module may be as follows: Figure 13 As shown. The third normalization layer can be a layer normalization layer. Figure 13 The FFN involved can be the aforementioned second feedforward layer. An exemplary structure of this second feedforward layer can be as follows: Figure 14 As shown. In Figure 14 The structure involved includes a reshape layer, a linear layer, and a dropout layer.

[0084] Specifically, obtaining multi-scale features through multi-scale enhanced window features and image features output by the second downsampling layer can include: obtaining second fusion features through multi-scale enhanced window features and image features output by the second downsampling layer; and obtaining multi-scale features through the second fusion features and the second feedforward layer.

[0085] This embodiment provides an image processing method that enables a neural network model to more accurately and comprehensively determine the reflection content that needs to be removed from the input image, thereby achieving better reflection content removal in the output image. In this embodiment, a multi-scale window is employed to effectively balance local and global information. This method achieves cross-scale feature aggregation and establishes long-range dependencies. To further control computational costs, this embodiment uses a spatial-channel association method with linear complexity to efficiently aggregate spatial and channel information from a hierarchical window.

[0086] Based on the foregoing embodiments, when the neural network model provided in this application embodiment simultaneously includes a first processing module, a second processing module, and a third processing module, the structure of the neural network model can be as follows: Figure 15 As shown.

[0087] exist Figure 15 In the neural network model shown, the first feature extraction modules enclosed by dashed box 30 can all be understood as the first feature extraction modules in the downsampling layer. The first feature extraction modules enclosed by dashed box 31 can all be understood as the first feature extraction modules in the upsampling layer. Figure 15 The example shown includes four downsampling layers and four upsampling layers. In each module, the value in the lower right corner of "n" indicates the number of modules. For example, if the first feature extraction module corresponds to "n2", it means that this downsampling layer includes two first feature extraction modules, and these two first feature extraction modules are sequentially connected. As another example, if the second feature extraction module corresponds to "n", it means... 10 The '' indicates that there are ten second feature extraction modules, and these ten second feature extraction modules are sequentially connected.

[0088] based on Figure 15 The structure shown includes one first feature extraction module in the first downsampling layer, two first feature extraction modules in the second downsampling layer, three first feature extraction modules in the third downsampling layer, and four first feature extraction modules in the fourth downsampling layer. Additionally, it includes six first feature extraction modules in the first upsampling layer, seven first feature extraction modules in the second upsampling layer, eight first feature extraction modules in the third upsampling layer, and nine first feature extraction modules in the fourth upsampling layer.

[0089] like Figure 15As shown, the neural network model involved in this embodiment can be a U-shaped architecture model, and in the case of adopting a single-stage U-shaped architecture, skip connections are also introduced (e.g., the connection method between the second feature extraction module and the third feature extraction module and the first feature extraction module). In this UNet structure, three basic modules are integrated: a non-linear module without activation function (first feature extraction module), an improved hierarchical Transformer module (third feature extraction module), and an improved fast Fourier transform and Transformer fusion module (fourth feature extraction module). Therefore, Figure 15 The architecture shown can be understood as a U-shaped Fast Fourier Transform Transformer and a hierarchical Transformer (F2T2-HiT) architecture.

[0090] based on Figure 15 In the structure shown, during the upsampling process, the global frequency domain features obtained by the second feature extraction module and the multi-scale features obtained by the third feature extraction module can be used in different upsampling layers. For example, if the second feature extraction module processes the features output from the second downsampling layer to obtain global frequency domain features, these global frequency domain features can be transmitted to the third upsampling layer for use during feature processing. Similarly, if the third feature extraction module processes the features output from the fourth downsampling layer to obtain multi-scale features, these multi-scale features can be transmitted to the first upsampling layer for use during feature processing. In this case, the first upsampling layer can perform upsampling based on the first image features and the multi-scale features, and then transmit the processed output to the second upsampling layer. The third upsampling layer will then further upsampling the output of the second upsampling layer based on the global frequency domain features.

[0091] Please see Figure 16 This application provides an image processing apparatus 400 for running a neural network model, the neural network model including multiple downsampling layers and multiple upsampling layers, and the method of apparatus 400 includes: The downsampling unit 410 is used to downsample the input image through multiple downsampling layers to obtain the first image features. In the multiple downsampling layers, the first downsampling layer processes the input image, and each subsequent downsampling layer processes the output of the adjacent preceding downsampling layer. The global frequency feature acquisition unit 420 is used to obtain global frequency domain features by using the frequency domain information and spatial information of the image features output by the first downsampling layer. The first downsampling layer is one of multiple downsampling layers. The multi-scale feature acquisition unit 430 is used to obtain multi-scale features through the image features output by the second downsampling layer and windows of multiple scales. The second downsampling layer is one of multiple downsampling layers. The upsampling unit 440 is used to upsample the first image features through multiple upsampling layers, global frequency domain features, and multi-scale features to obtain an output image with reflection content removed.

[0092] In one approach, the downsampling layer includes at least one first feature extraction module, which includes a first normalization layer, a convolutional layer, and a simplified channel attention layer. The downsampling unit 410 is specifically used to process the image features input to the first feature extraction module through the first normalization layer, and then transmit the image features output by the first normalization layer to the convolutional layer; after processing the input image features in the convolutional layer, the image features output by the convolutional layer are transmitted to the simplified channel attention layer; after processing the input image features in the simplified channel attention layer, the output of the first feature extraction module is obtained by combining the output of the simplified channel attention layer and the image features input to the first feature extraction module.

[0093] As one approach, the neural network model also includes a second feature extraction module.

[0094] The global frequency feature acquisition unit 420 is specifically used to acquire the frequency domain features and spatial features of the image features output by the first downsampling layer through the second feature extraction module. The frequency domain features represent the frequency domain information of the image features output by the first downsampling layer, and the spatial features represent the spatial information of the image features output by the first downsampling layer. The frequency domain features and spatial features are fused through the second feature extraction module to obtain the global frequency domain features.

[0095] Optionally, the second feature extraction module includes a Fourier feature extraction layer, which comprises a spatial feature extraction module, a frequency domain feature extraction module, and a fusion module. The global frequency feature acquisition unit 420 is specifically used to acquire spatial features from the image features output by the first downsampling layer through the spatial feature extraction module, acquire frequency domain features from the image features output by the first downsampling layer through the frequency domain feature extraction module, and fuse the frequency domain features and spatial features through the fusion module to obtain global frequency domain features.

[0096] Optionally, the second feature extraction module further includes a second normalization layer and a first feedforward layer. The global frequency feature acquisition unit 420 is specifically used to process the image features output by the first downsampling layer through the second normalization layer to obtain second image features; to obtain the spatial features of the second image features through the spatial feature extraction module; and to obtain the frequency domain features of the second image features through the frequency domain feature extraction module. The frequency domain features and spatial features are then fused through the fusion module to obtain a first fused feature. Finally, the global frequency domain features are obtained through the image features output by the first feedforward layer and the first downsampling layer, and the first fused feature.

[0097] As one approach, the neural network model also includes a third feature extraction module; the third feature extraction module includes a hierarchical window self-attention layer, which includes a dual-domain feature extraction layer, a spatial autocorrelation layer, and a channel autocorrelation layer.

[0098] In this manner, the multi-scale feature acquisition unit 430 is used to process the image features output by the second downsampling layer through the dual-domain feature extraction layer to obtain dual-domain features; and to process the dual-domain features through windows of multiple scales, spatial autocorrelation layers and channel autocorrelation layers to obtain multi-scale features.

[0099] Optionally, the multi-scale feature acquisition unit 430 is specifically used to process the dual-domain features through windows of multiple scales to obtain window features of each window at multiple scales; process the window features through spatial autocorrelation layer and channel autocorrelation layer to obtain enhanced window features of each window at multiple scales; concatenate the enhanced window features corresponding to each window at multiple scales to obtain multi-scale enhanced window features; and obtain multi-scale features through multi-scale enhanced window features and image features output by the second downsampling layer.

[0100] Optionally, the third feature extraction module also includes a third normalization layer and a second feedforward layer. The multi-scale feature acquisition unit 430 is specifically used to obtain a second fused feature by using the multi-scale enhanced window feature and the image feature output by the second downsampling layer; and to obtain multi-scale features by using the second fused feature and the second feedforward layer.

[0101] This application provides an image processing apparatus that enables a neural network model to more accurately and comprehensively determine the reflection content that needs to be removed from the input image, thereby making the output image have a better reflection content removal effect.

[0102] It should be noted that the device embodiments in this application correspond to the aforementioned method embodiments. The specific principles in the device embodiments can be found in the content of the aforementioned method embodiments, and will not be repeated here.

[0103] The following will combine Figure 17 This application describes an electronic device.

[0104] Please see Figure 17 Based on the aforementioned image processing method, image generation method, and apparatus, this application embodiment also provides another electronic device 100 capable of executing the aforementioned image processing method. The electronic device 2000 includes one or more (only one shown in the figure) processors 202, a memory 204, a network module 206, a sensor module 208, and an audio acquisition device 210, all coupled to each other. The memory 204 stores programs capable of executing the contents of the aforementioned embodiments, and the processor 202 can execute the programs stored in the memory 204.

[0105] The processor 202 may include one or more processing cores. The processor 202 connects to various parts of the electronic device 2000 via various interfaces and lines, and performs various functions and processes data of the electronic device 2000 by running or executing instructions, programs, code sets, or instruction sets stored in the memory 204, and by calling data stored in the memory 204. Optionally, the processor 202 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 202 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 202 and may be implemented separately using a communication chip.

[0106] The memory 204 may include random access memory (RAM) or read-only memory (ROM). The memory 204 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 204 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described below.

[0107] Network module 206 is used to enable information interaction between electronic device 2000 and other devices, such as transmitting device control commands, manipulation request commands, and status information acquisition commands. However, the specific network module 206 may differ depending on whether the electronic device 2000 is a different device.

[0108] Sensor module 208 may include at least one sensor. Specifically, sensor module 208 may include, but is not limited to, a light sensor, a motion sensor, a pressure sensor, an infrared thermal sensor, a distance sensor, an acceleration sensor, and other sensors.

[0109] The pressure sensor is a sensor that detects pressure generated by pressing on the electronic device 2000. That is, the pressure sensor detects pressure generated by contact or pressing between the user and the electronic device, such as the pressure generated by contact or pressing between the user's ear and the mobile terminal. Therefore, the pressure sensor can be used to determine whether contact or pressing has occurred between the user and the electronic device 2000, and the magnitude of the pressure.

[0110] The accelerometer can detect the magnitude of acceleration in various directions (generally three axes), and when stationary, it can detect the magnitude and direction of gravity. It can be used for applications such as identifying the posture of electronic devices 2000 (e.g., screen orientation switching, related games, magnetometer posture calibration), and vibration recognition functions (e.g., pedometer, tapping). In addition, electronic devices 2000 can also be equipped with other sensors such as gyroscopes, barometers, hygrometers, and thermometers, which will not be elaborated upon here.

[0111] Audio acquisition device 210 is used for acquiring audio signals. Optionally, audio acquisition device 210 includes multiple audio acquisition devices. These audio acquisition devices can be microphones. For example, in one embodiment, audio acquisition device 210 may include two microphones, in which one microphone may correspond to one analog-to-digital converter (ADC), and the other microphone may correspond to two ADCs with different analog gains. In another embodiment, audio acquisition device 210 may include three microphones. In this embodiment, two microphones (e.g., a main microphone and a secondary microphone) may each correspond to one ADC, and the other microphone (e.g., a camera microphone) may correspond to two ADCs with different analog gains.

[0112] In one approach, the network module of the electronic device 2000 is a radio frequency (RF) module. This RF module is used to receive and transmit electromagnetic waves, converting electromagnetic waves into electrical signals, thereby enabling communication with communication networks or other devices. The RF module may include various existing circuit elements used to perform these functions, such as antennas, RF transceivers, digital signal processors, encryption / decryption chips, user identity module (SIM) cards, memory, etc. For example, the RF module can interact with external devices by transmitting or receiving electromagnetic waves, thereby receiving audio signals transmitted by external devices.

[0113] Furthermore, the electronic device 2000 may also include an image acquisition device for image acquisition. For example, the image acquisition device can be used to capture video, still images, or moving images.

[0114] Please refer to Figure 18 This diagram illustrates a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. The computer-readable medium 800 stores program code that can be called by a processor to execute the methods described in the above method embodiments.

[0115] The computer-readable storage medium 800 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 800 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 800 has storage space for program code 810 that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 810 may be compressed, for example, in a suitable form.

[0116] In summary, the image processing method, apparatus, and electronic device provided in this application firstly downsample the input image through multiple downsampling layers to obtain a first image feature. Furthermore, a global frequency domain feature can be obtained using the frequency domain and spatial information of the image features output by the first downsampling layer, where the first downsampling layer is one of the multiple downsampling layers. Then, a multi-scale feature is obtained using the image features output by a second downsampling layer and windows of multiple scales, where the second downsampling layer is one of the multiple downsampling layers. Finally, the first image feature is upsampled using the multiple upsampling layers, the global frequency domain feature, and the multi-scale feature to obtain an output image with reflections removed. Thus, by means of the above method, when downsampling the input image, global frequency domain features are obtained through the frequency domain information and spatial information of the image features output by the first downsampling layer, and multi-scale features are obtained through the image features output by the second downsampling layer and windows of multiple scales. Therefore, during the upsampling process to obtain the output image, the obtained global frequency domain features and multi-scale features can be combined to enable the neural network model to more accurately and comprehensively determine the reflection content that needs to be removed from the input image, so that the output image has a better reflection content removal effect.

[0117] In one approach, the solution proposed in this application offers several technical advantages, significantly improving reflection removal performance. By employing Fast Fourier Convolutions (FFCs), a wide receptive field is achieved, enabling comprehensive feature extraction and effectively capturing local and global dependencies. The F2T2 block (second feature extraction module) integrates the FFT mechanism into the Transformer architecture, forming a hybrid structure that simultaneously processes the spatial and frequency domains, enhancing the neural network model's ability to handle diverse reflection types. This dual-domain processing approach, combined with multi-scale receptive field modeling, offers superior performance when handling high-resolution and complex reflection scenes. Furthermore, depthwise convolution and spatial-channel correlation methods improve computational efficiency, making the system suitable for real-time applications. The model's robust design ensures good generalization ability under various reflection conditions, resulting in better performance in practical applications such as photography, security, automotive, and augmented reality. Overall, this solution achieves excellent reflection removal results by solving the problem of large-area complex reflections while maintaining high efficiency and versatility.

[0118] The single-image reflection removal method provided in this application is highly applicable to a variety of practical application scenarios. For example, in the field of photography, this method can effectively improve image quality and easily eliminate excess reflections on glass windows in landscape, architectural, and product photography; in security and intelligent monitoring, this method can improve the clarity of video images captured through windows, ensuring clearer monitoring results; and in automotive applications, this method helps eliminate windshield reflections, thereby providing more accurate scene analysis for autonomous driving and driver assistance systems. Furthermore, in the fields of augmented reality (AR) and smart glasses, this method can significantly improve the user experience by removing interfering reflections from transparent surfaces. Whether for professional image production, safety-critical systems, or immersive AR environments, the method provided in this application can ensure clear, high-quality visual effects in various complex scenarios.

[0119] In the embodiments of this application, besides using Fast Fourier Convolution, other techniques can also be used to achieve a wide receptive field. For example, dilated convolution or attention mechanisms can be employed to achieve a wider receptive field. Furthermore, in addition to combining a dual-domain hybrid structure of spatial and frequency domains to simultaneously capture local and global features, multi-scale convolution or combining traditional convolutional layers with self-attention mechanisms can also be considered to simultaneously capture local and global features. Additionally, models pre-trained on reflection removal tasks can be leveraged to fine-tune for different reflection scenarios (e.g., the aforementioned practical applications such as photography, security vehicles, and augmented reality) to reduce the complexity of multi-domain processing.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An image processing method, characterized by, Applied to a neural network model, wherein the neural network model includes multiple downsampling layers and multiple upsampling layers, the method includes: The input image is downsampled through the multiple downsampling layers to obtain the first image features. In the multiple downsampling layers, the first downsampling layer processes the input image, and each subsequent downsampling layer processes the output of the adjacent preceding downsampling layer. Global frequency domain features are obtained by using the frequency domain information and spatial information of the image features output by the first downsampling layer, where the first downsampling layer is one of the plurality of downsampling layers; Multi-scale features are obtained by using the image features output by the second downsampling layer and windows of multiple scales, wherein the second downsampling layer is one of the multiple downsampling layers; The first image features are upsampled using the multiple upsampling layers, the global frequency domain features, and the multi-scale features to obtain an output image with reflections removed.

2. The method of claim 1, wherein, The downsampling layer includes one or more first feature extraction modules. Each first feature extraction module includes a first normalization layer, a convolutional layer, and a simplified channel attention layer. The processing procedure of the first feature extraction module includes: The first normalization layer processes the image features input to the first feature extraction module, and the image features output by the first normalization layer are transmitted to the convolutional layer. After the convolutional layer processes the input image features, the output image features of the convolutional layer are transmitted to the simplified channel attention layer. After the simplified channel attention layer processes the input image features, the output of the first feature extraction module is obtained by combining the output of the simplified channel attention layer with the image features input to the first feature extraction module.

3. The method of claim 2, wherein, In the case where the downsampling layer includes multiple first feature extraction modules, the output of the preceding first feature extraction module serves as the input of the adjacent following first feature extraction module.

4. The method of claim 1, wherein, The neural network model further includes a second feature extraction module, which obtains global frequency domain features by combining the frequency domain information and spatial information of the image features output from the first downsampling layer, including: The second feature extraction module obtains the frequency domain features and spatial features of the image features output by the first downsampling layer. The frequency domain features represent the frequency domain information of the image features output by the first downsampling layer, and the spatial features represent the spatial information of the image features output by the first downsampling layer. The second feature extraction module fuses the frequency domain features and spatial features to obtain global frequency domain features.

5. The method according to claim 4, characterized in that, The second feature extraction module includes a Fourier feature extraction layer, which comprises a spatial feature extraction module, a frequency domain feature extraction module, and a fusion module. The step of obtaining frequency domain features and spatial features from the image features output by the first downsampling layer through the second feature extraction module includes: The spatial feature extraction module obtains the spatial features in the image features output by the first downsampling layer, and the frequency domain feature extraction module obtains the frequency domain features in the image features output by the first downsampling layer. The step of fusing the frequency domain features and spatial features through the second feature extraction module to obtain global frequency domain features includes: The frequency domain features and spatial features are fused by the fusion module to obtain global frequency domain features.

6. The method according to claim 5, characterized in that, The second feature extraction module further includes a second normalization layer and a first feedforward layer, and the method further includes: The second normalization layer processes the image features output by the first downsampling layer to obtain the second image features; The step of obtaining spatial features from the image features output by the first downsampling layer through the spatial feature extraction module and obtaining frequency domain features from the image features output by the first downsampling layer through the frequency domain feature extraction module includes: obtaining spatial features of the second image feature through the spatial feature extraction module and obtaining frequency domain features of the second image feature through the frequency domain feature extraction module. The process of fusing the frequency domain features and spatial features through the fusion module to obtain global frequency domain features includes: The frequency domain features and spatial features are fused by the fusion module to obtain a first fused feature; the global frequency domain features are obtained by combining the image features output by the first feedforward layer and the first downsampling layer with the first fused feature.

7. The method according to claim 1, characterized in that, The neural network model further includes a third feature extraction module; the third feature extraction module includes a hierarchical window self-attention layer, which includes: a dual-domain feature extraction layer, a spatial autocorrelation layer, and a channel autocorrelation layer; the image features output by the second downsampling layer and windows of multiple scales are used to obtain multi-scale features, including: The image features output by the second downsampling layer are processed by the dual-domain feature extraction layer to obtain dual-domain features; The dual-domain features are processed by windows of multiple scales, the spatial autocorrelation layer, and the channel autocorrelation layer to obtain multi-scale features.

8. The method according to claim 7, characterized in that, The process of processing the dual-domain features through windows of multiple scales, the spatial autocorrelation layer, and the channel autocorrelation layer to obtain multi-scale features includes: The dual-domain features are processed by windows of various scales to obtain window features of each scale. The spatial autocorrelation layer and the channel autocorrelation layer are used to process the features of each window to obtain the enhanced window features of windows at various scales. The enhancement window features corresponding to windows of multiple scales are concatenated to obtain multi-scale enhancement window features; Multi-scale features are obtained by combining the multi-scale enhanced window features and the image features output by the second downsampling layer.

9. The method according to claim 8, characterized in that, The third feature extraction module further includes a third normalization layer and a second feedforward layer. The multi-scale features obtained through the multi-scale enhanced window features and the image features output by the second downsampling layer include: The second fusion feature is obtained by combining the multi-scale enhanced window features and the image features output by the second downsampling layer. Multi-scale features are obtained through the second fusion feature and the second feedforward layer.

10. An image processing apparatus, characterized in that, For running a neural network model, the neural network model including multiple downsampling layers and multiple upsampling layers, the device includes: The downsampling unit is used to downsample the input image through the multiple downsampling layers to obtain the first image feature. In the multiple downsampling layers, the first downsampling layer processes the input image, and each subsequent downsampling layer processes the output of the adjacent preceding downsampling layer. The global frequency feature acquisition unit is used to obtain global frequency domain features through the frequency domain information and spatial information of the image features output by the first downsampling layer, wherein the first downsampling layer is one of the plurality of downsampling layers; A multi-scale feature acquisition unit is used to obtain multi-scale features from the image features output by the second downsampling layer and windows of multiple scales, wherein the second downsampling layer is one of the multiple downsampling layers; The upsampling unit is used to upsample the first image features through the multiple upsampling layers, the global frequency domain features, and the multi-scale features to obtain an output image with reflection content removed.

11. An electronic device, characterized in that, It includes a processor and a memory; one or more programs are stored in the memory and configured to be executed by the processor to implement the method of any one of claims 1-9.

12. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the method as described in any one of claims 1-9.

Images