Image denoising method and device based on CLIP image encoder and fourier structure encoder, equipment and medium

Abstract

This application relates to an image denoising method, apparatus, device, and medium based on a CLIP image encoder and a Fourier structure encoder. The method includes: uniformly dividing a fused feature map into multiple sub-feature groups along the channel dimension; for each sub-feature group, concatenating all other sub-feature groups along the channel dimension to form complementary context features; predicting affine parameters from the complementary context features using an affine transformation module; performing affine modulation on the current sub-feature group; recombining all affine-modulated sub-feature groups; smoothing the result using a 3×3 convolutional layer to output a refined feature map; calling a preset decoder to restore the original image resolution of the RGB image to be denoised layer by layer, obtaining high-dimensional feature data matching the original image resolution; and mapping the high-dimensional feature data to the denoised RGB image using a 3×3 convolutional layer. This application solves the problems of incomplete single-domain feature representation and semantic-detail imbalance in traditional image denoising methods.

Description

Technical Field

[0001] This application relates to the field of image denoising, and in particular to an image denoising method, corresponding apparatus, electronic device and computer-readable storage medium based on CLIP image encoder and Fourier structure encoder. Background Technology

[0002] Image denoising is a fundamental and core task in computational photography, multimedia systems, and computer vision. Its core objective is to accurately recover visually clear, detailed, and semantically consistent clean images from observation images contaminated by sensor noise, environmental interference, and other factors. With the widespread adoption of smartphones, portable imaging devices, and machine vision systems, scenarios with significant noise, such as low light, high ISO, and rapid shooting, have become mainstream application scenarios. Developing efficient, robust, and universal image denoising algorithms has significant practical value for improving the user experience of terminal imaging and ensuring the accuracy of machine vision tasks.

[0003] In recent years, deep learning-based image denoising methods have made breakthrough progress due to their end-to-end feature learning capabilities, becoming the mainstream research in this field. Early classic methods such as DnCNN achieved excellent performance in the task of removing synthetic additive white Gaussian noise (AWGN) through residual learning and batch normalization mechanisms; subsequently, FFDNet further introduced an adjustable noise level map, improving the model's adaptability to synthetic noise of different intensities. However, the core limitation of such methods lies in modeling synthetic noise. The pre-set noise distribution is fundamentally different from the complex noise characteristics of the camera imaging process in the real world, resulting in a significant decrease in the model's generalization ability on real-world images.

[0004] Real-world image noise originates from the entire camera sensing and signal processing chain, encompassing multiple noise sources such as photon shot noise, sensor readout noise, color filter array interpolation errors, and nonlinear transformations of the camera's internal image signal processor (ISP). This results in complex characteristics of real-world noise, including signal dependence, spatial heterogeneity, channel correlation, and device specificity. Its statistical distribution cannot be accurately described by simple parametric models (such as AWGN or Poisson noise). This "domain difference" problem causes denoising models trained on synthetic noise datasets to experience sharp performance drops, detail blurring, artifact introduction, and texture distortion when applied to real images, making them unsuitable for practical applications.

[0005] To alleviate the challenge of denoising real-world noise, researchers have primarily explored two core directions: noise modeling and data synthesis, and network architecture and adaptive mechanism design. While some progress has been made, significant technical shortcomings remain, including:

[0006] Firstly, methods based on real noise modeling and data synthesis attempt to simulate the ISP processing flow of commercial cameras through reverse engineering, constructing a realistic noise synthesis model from the RAW domain to the sRGB domain. This generates large-scale paired real noise-clean image training data, addressing the problem of scarce real data. For example, CBDNet improves its adaptability to real noise by jointly estimating noise levels and denoising results; CycleISP uses cycle consistency adversarial learning to generate noise image pairs that more closely resemble the real distribution. However, due to the proprietary and black-box nature of commercial camera ISP pipelines, existing technologies cannot accurately model the coupling relationships of all noise sources. The synthesized noise distribution still inevitably deviates from the noise in the real scene, limiting the model's generalization upper limit. Furthermore, the complex noise synthesis process significantly increases the cost of data preparation.

[0007] Secondly, deep denoising networks based on architectural design and adaptive mechanisms focus on designing more expressive network architectures. By enhancing the model's feature extraction and adaptive capabilities, they improve the generalization ability to unknown real noise. For example, MIRNet expands the model's receptive field through multi-scale feature fusion, strengthening its ability to capture noise and details at different scales; Uformer and Restormer introduce the Transformer's self-attention mechanism into low-level visual tasks, effectively capturing long-range dependencies in images; dynamic filtering networks, represented by Involution, dynamically generate convolutional kernels based on the input image content, achieving adaptive modulation of features in different regions. However, these methods still have significant shortcomings: First, adaptive mechanisms such as dynamic filtering are often accompanied by high computational costs and low parameter efficiency, making them difficult to deploy on resource-constrained terminal devices such as smartphones and drones; second, most methods are still limited to feature interactions in a single spatial domain, failing to fully integrate frequency domain structural awareness information and high-level semantic priors, resulting in insufficient model recovery of image structural details or loss of semantic consistency during denoising.

[0008] Currently, some research attempts to introduce pre-trained visual-language models (such as CLIP) into image denoising tasks, leveraging their high-level semantic priors learned on large-scale cross-modal data to improve the model's robustness to noise. For example, some methods inject frozen CLIP features into the decoder, mitigating the problem of detail loss during denoising through semantic guidance. However, the optimization goal of CLIP features is high-level semantic recognition, rather than low-level pixel-level detail restoration. They lack sensitivity to structural information such as local image texture and edge contours, and direct introduction can easily lead to overly smooth denoising results and high-frequency structural distortion, making it difficult to achieve a balance between semantic fidelity and detail restoration.

[0009] Existing real-image denoising techniques still suffer from the following technical shortcomings, including:

[0010] Firstly, the generalization ability is insufficient: the model is prone to overfitting to the noise distribution of the training data, has poor adaptability to complex real noise generated by unknown shooting equipment and unknown scenes, and has poor domain transfer performance.

[0011] Secondly, efficiency and performance are difficult to balance: dynamic filtering, self-attention and other highly expressive mechanisms often bring high computational overhead, making it difficult to meet the real-time and low-power requirements of terminal devices while ensuring denoising performance.

[0012] Third, the feature representation is incomplete: existing methods fail to effectively integrate high-level semantic priors, spatial domain detail features and frequency domain structural information, which leads to problems such as semantic distortion, structural ambiguity or artifact introduction in the denoising results, and cannot take into account both semantic consistency and texture fidelity.

[0013] In summary, existing real-image denoising techniques suffer from problems such as insufficient generalization ability, difficulty in balancing efficiency and performance, and incomplete feature representation. The applicant has made corresponding explorations to address these issues. Summary of the Invention

[0014] The purpose of this application is to solve the above-mentioned problems by providing an image denoising method, corresponding apparatus, electronic device and computer-readable storage medium based on CLIP image encoder and Fourier structure encoder.

[0015] To achieve the various objectives of this application, the following technical solution is adopted:

[0016] An image denoising method based on a CLIP image encoder and a Fourier structure encoder, proposed to meet one of the purposes of this application, includes:

[0017] Obtain the RGB image to be denoised, which contains real noise, to determine the semantic stream input data and structural stream input data corresponding to the RGB image to be denoised;

[0018] The semantic stream input data is input to a parameter-frozen, pre-trained CLIP image encoder to extract a noise-robust global semantic feature map; the structure stream input data is input to a learnable Fourier structure encoder isomorphic to the CLIP image encoder to extract a spatial structure feature map aligned with the spatial and channel dimensions of the global semantic feature map.

[0019] For each level of global semantic feature map and spatial structure feature map, the spatial structure feature map is used as the generation condition. A channel dynamic filter is generated based on global average pooling in the spatial dimension. A spatial dynamic filter is generated by concatenating global average pooling and global max pooling in the channel dimension. The channel dynamic filter and the spatial dynamic filter are concatenated and normalized to obtain a dynamic convolution kernel. The dynamic convolution kernel is used to adaptively convolve and modulate the global semantic feature map of the corresponding level to output a fused feature map.

[0020] The fused feature map is uniformly divided into multiple sub-feature groups along the channel dimension. For each sub-feature group, all other sub-feature groups are concatenated along the channel dimension to form complementary context features. The affine transformation module predicts affine parameters from the complementary context features to perform affine modulation on the current sub-feature group. All affine-modulated sub-feature groups are recombined and smoothed by a convolutional layer with a kernel size of 3×3 to output a refined feature map.

[0021] The preset decoder is invoked to sequentially perform feature upsampling, symmetric skip connection fusion, and residual convolution refinement on the refined feature map, restoring it layer by layer to the original image resolution of the RGB image to be denoised, thereby obtaining high-dimensional feature data that matches the original image resolution. The high-dimensional feature data is then mapped to the denoised RGB image through a convolutional layer with a kernel size of 3×3.

[0022] Optionally, the step of determining the semantic stream input data corresponding to the RGB image to be denoised includes:

[0023] Obtain the RGB image to be denoised, which contains real noise, and perform normalization processing on the RGB image to be denoised in the same way as in the CLIP image encoder pre-training stage.

[0024] Linear pixel value transformation is performed on the red, green, and blue color channels of the RGB image to be denoised. The original pixel value of each color channel is subtracted from the mean pixel value of that color channel, and then the mean-free pixel value is divided by the standard deviation of the pixel value of that color channel to generate semantic stream input data that retains the original color and appearance information of the RGB image to be denoised.

[0025] Optionally, the step of determining the structured stream input data corresponding to the RGB image to be denoised includes:

[0026] Obtain the RGB image to be denoised, which contains real noise, and separate the RGB image to be denoised into single-color channel images corresponding to the red, green, and blue color channels along the channel dimension;

[0027] A two-dimensional discrete Fourier transform is performed on each single-color channel image to obtain its corresponding frequency domain representation. The phase spectrum representing the image structure layout is decoupled from the frequency domain representation, and the amplitude spectrum representing the image intensity information in the frequency domain representation is set to a constant.

[0028] The phase spectrum obtained by decoupling is inverted, and then the inverted phase spectrum is subjected to a two-dimensional discrete Fourier inverse transform to obtain the phase inverted image corresponding to each single color channel image.

[0029] The phase-reversed images corresponding to the red, green, and blue color channels are re-stitched along the channel dimensions to generate structured stream input data that highlights the geometric structure, edges, and contour information of the image.

[0030] Optionally, for each level's global semantic feature map and spatial structure feature map, using the spatial structure feature map as the generation condition, a channel dynamic filter is generated based on global average pooling in the spatial dimension. A spatial dynamic filter is generated by concatenating global average pooling and global max pooling in the channel dimension. The channel dynamic filter and the spatial dynamic filter are concatenated and normalized to obtain a dynamic convolution kernel. The dynamic convolution kernel is then used to adaptively convolve and modulate the corresponding level's global semantic feature map to output a fused feature map. This step includes:

[0031] For each level of global semantic feature map and spatial structure feature map obtained after feature extraction of the RGB image to be denoised, the spatial structure feature map is used as the generation condition to perform global average pooling of the spatial dimension to generate a channel dynamic filter.

[0032] The spatial structure feature map is simultaneously subjected to global average pooling and global max pooling in the channel dimension to obtain a first pooling result and a second pooling result. The first pooling result and the second pooling result are then concatenated along the channel dimension to generate a spatial dynamic filter.

[0033] The channel dynamic filter and the spatial dynamic filter are concatenated along the channel dimension to obtain the original dynamic convolution kernel. The original dynamic convolution kernel is then normalized to obtain the dynamic convolution kernel.

[0034] The dynamic convolution kernel performs element-wise weighted summation and feature modulation on the pixel regions of the global semantic feature map at the corresponding level, embedding spatial structure features into the global semantic features, and outputting a fused feature map that integrates global semantic information and spatial structure details.

[0035] Optionally, the steps of uniformly dividing the fused feature map into multiple sub-feature groups along the channel dimension, concatenating all other sub-feature groups along the channel dimension into complementary context features for each sub-feature group, predicting affine parameters from the complementary context features using an affine transformation module to perform affine modulation on the current sub-feature group, recombining all affine-modulated sub-feature groups, and smoothing them through a 3×3 convolutional layer to output a refined feature map include:

[0036] The fused feature map, which integrates global semantic information and spatial structural details, is uniformly divided along the channel dimension to obtain multiple sub-feature groups with consistent channel dimensions.

[0037] For each sub-feature group, all other sub-feature groups except the current sub-feature group are concatenated along the channel dimension to generate complementary context features corresponding to the current sub-feature group;

[0038] The complementary context features are input into the affine transformation module, which predicts the affine parameters for the current sub-feature group. The affine parameters are then used to perform affine modulation processing on the current sub-feature group.

[0039] All affine-modulated sub-feature groups are reassembled along the channel dimension. The reassembled feature map is then input into a 3×3 convolutional layer for spatial smoothing, and the refined feature map with channel interaction refinement is output.

[0040] Optionally, a preset decoder is invoked to sequentially perform feature upsampling, symmetric skip connection fusion, and residual convolution refinement on the refined feature map, restoring it layer by layer to the original image resolution of the RGB image to be denoised, thereby obtaining high-dimensional feature data matching the original image resolution. The high-dimensional feature data is then mapped to the denoised RGB image through a convolutional layer with a kernel size of 3×3. This step includes:

[0041] Construct a decoder that is structurally symmetrical to the CLIP image encoder and the Fourier structure encoder, and input the refined feature map into the decoder;

[0042] The refined feature map is sequentially subjected to feature upsampling, symmetric skip connection fusion, and residual convolution refinement at each level of the decoder, gradually restoring the spatial resolution of the feature map to the original image resolution of the RGB image to be denoised, thereby obtaining high-dimensional feature data that matches the original image resolution;

[0043] The high-dimensional feature data is input into a convolutional layer with a kernel size of 3×3. The convolutional layer performs feature dimension mapping data processing to map the high-dimensional feature data into a denoised RGB image.

[0044] Optionally, all weight parameters of the CLIP image encoder remain frozen throughout the feature extraction process and do not participate in gradient calculation and parameter updates.

[0045] The number of network layers, number of channels, and downsampling strategy of the Fourier structure encoder are completely isomorphic to those of the CLIP image encoder.

[0046] An image denoising apparatus based on a CLIP image encoder and a Fourier structure encoder, provided to suit another purpose of this application, comprises:

[0047] The data acquisition module is configured to acquire the RGB image to be denoised, which contains real noise, in order to determine the semantic stream input data and structural stream input data corresponding to the RGB image to be denoised.

[0048] The feature map extraction module is configured to input the semantic stream input data into a parameter-frozen, pre-trained CLIP image encoder to extract a noise-robust global semantic feature map; and input the structure stream input data into a learnable Fourier structure encoder isomorphic to the CLIP image encoder to extract a spatial structure feature map aligned with the spatial and channel dimensions of the global semantic feature map.

[0049] The fusion feature map construction module is configured to generate channel dynamic filters based on global average pooling in the spatial dimension, using the spatial structure feature map as the generation condition, and then generate spatial dynamic filters by concatenating global average pooling and global max pooling in the channel dimension. The channel dynamic filters and the spatial dynamic filters are then concatenated and normalized to obtain dynamic convolution kernels. The dynamic convolution kernels are used to adaptively convolve and modulate the global semantic feature maps of the corresponding layers to output fusion feature maps.

[0050] The refined feature map construction module is configured to uniformly divide the fused feature map into multiple sub-feature groups along the channel dimension. For each sub-feature group, all other sub-feature groups are concatenated along the channel dimension to form complementary context features. The affine transformation module predicts affine parameters from the complementary context features to perform affine modulation on the current sub-feature group. All affine-modulated sub-feature groups are recombined and smoothed by a convolutional layer with a kernel size of 3×3 to output the refined feature map.

[0051] The image denoising module is configured to call a preset decoder to sequentially perform feature upsampling, symmetric skip connection fusion, and residual convolution refinement on the refined feature map, restoring it layer by layer to the original image resolution of the RGB image to be denoised, thereby obtaining high-dimensional feature data that matches the original image resolution. The high-dimensional feature data is then mapped to the denoised RGB image through a convolutional layer with a kernel size of 3×3.

[0052] An electronic device provided for another purpose of this application includes a central processing unit and a memory, wherein the central processing unit is configured to invoke and run a computer program stored in the memory to perform the steps of the image denoising method based on CLIP image encoder and Fourier structure encoder described in this application.

[0053] A computer-readable storage medium is provided for another purpose of this application, which stores, in the form of computer-readable instructions, a computer program implemented according to the image denoising method based on the CLIP image encoder and Fourier structure encoder, which, when called by a computer, executes the steps included in the corresponding method.

[0054] Compared to existing technologies, this application addresses the problems of insufficient generalization ability, difficulty in balancing efficiency and performance, and incomplete feature representation in existing real-image denoising techniques. This application provides, but is not limited to, the following beneficial effects:

[0055] Firstly, this application introduces a parameter-frozen pre-trained CLIP image encoder to extract global semantic features. The distortion invariance obtained by CLIP during training on massive image and text data makes its extracted features naturally robust to sensor distortions such as noise and blur, allowing the model to avoid overfitting to specific training noise distributions. Simultaneously, combined with Fourier embedding for structural feature extraction, it can adapt to complex real-world noise generated by unknown shooting devices and extreme shooting environments without needing to accurately model the complex distribution of real-world noise. This effectively solves the "domain difference" problem of traditional methods and significantly improves the model's cross-domain generalization performance in real-world scenarios.

[0056] Secondly, this application achieves the separation, extraction, and precise fusion of semantic priors and structural details through a dual-stream architecture: the CLIP image encoder provides a stable global semantic prior, ensuring the semantic consistency of the denoising results and avoiding problems such as artifacts and semantic distortion; the Fourier structure encoder extracts spatial structural features from the structural flow data that enhances geometric structures, accurately capturing high-frequency details such as edges and textures that are submerged by noise. Simultaneously, by using dynamic convolutional kernels, structural information is finely embedded into semantic features, achieving adaptive modulation of semantic priors by structural details. This allows the denoising results to effectively remove noise while maximizing the recovery of sharp edges and rich textures in the image, solving the problems of incomplete single-domain feature representation and semantic-detail imbalance in traditional methods.

[0057] Thirdly, the adaptive mechanism of this application adopts a lightweight design, effectively solving the problems of high computational cost and low parameter efficiency in traditional dynamic filtering methods. On the one hand, it dynamically generates lightweight convolution kernels based on structural features, performing pixel-level adaptive modulation on semantic features, thus avoiding the huge parameter overhead of traditional dynamic convolution. On the other hand, it achieves inter-channel interaction by using channel grouping and lightweight affine transformation, completing feature refinement only through simple 1×1 convolution, element-wise operations, and 3×3 convolution smoothing, resulting in extremely low computational cost. While maintaining top-tier denoising performance, the overall model has a small model size and fast inference speed, meeting the power consumption and real-time requirements of resource-constrained terminal devices such as smartphones, drones, and surveillance cameras, making it easier to implement in practical applications. Attached Figure Description

[0058] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0059] Figure 1 This is a flowchart illustrating the image denoising method based on CLIP image encoder and Fourier structure encoder in the embodiments of this application.

[0060] Figure 2 This is a diagram showing the overall architecture of the dual-stream adaptive image denoising network based on CLIP image encoder and Fourier structure encoder in the embodiments of this application.

[0061] Figure 3 This is a schematic diagram illustrating the internal structure of the decoupled embedding module (DEM) and dynamic convolution modulation in the embodiments of this application.

[0062] Figure 4 This is a flowchart illustrating the workflow of the Inter-Channel Interaction Module (MCI) and the Affine Transformation Module (ATM) in the embodiments of this application.

[0063] Figure 5 This is a schematic block diagram of the image denoising device based on CLIP image encoder and Fourier structure encoder in the embodiments of this application;

[0064] Figure 6 This is a schematic diagram of the structure of the computer device in the embodiments of this application. Detailed Implementation

[0065] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0066] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0067] Those skilled in the art will understand that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0068] Those skilled in the art will understand that the terms "client," "terminal," and "terminal device" as used herein include both devices that receive wireless signals, devices that only possess wireless signal receiver capabilities without transmission capabilities, and devices with receiving and transmitting hardware, devices that have receiving and transmitting hardware capable of bidirectional communication over a bidirectional communication link. Such devices may include: cellular or other communication devices such as personal computers or tablets, having single-line displays, multi-line displays, or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service) that can combine voice, data processing, fax, and / or data communication capabilities; PDAs (Personal Digital Assistants) that may include radio frequency receivers, pagers, internet / intranet access, web browsers, notebooks, calendars, and / or GPS (Global Positioning System) receivers; and conventional laptops and / or handheld computers or other devices that have and / or include radio frequency receivers. As used herein, "client," "terminal," and "terminal device" can be portable, transportable, installed in a means of transportation (air, sea, and / or land), or suitable and / or configured to operate locally and / or in a distributed manner, operating in any other location on Earth and / or in space. "Client," "terminal," and "terminal device" as used herein can also be a communication terminal, an internet access terminal, or a music / video playback terminal, such as a PDA, a MID (Mobile Internet Device), and / or a mobile phone with music / video playback capabilities, or a smart TV, set-top box, etc.

[0069] The hardware referred to by the names "server," "client," and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann architecture, such as a central processing unit (including an arithmetic logic unit and a control unit), memory, input devices, and output devices. The computer program is stored in its memory, and the central processing unit loads the program stored in the secondary storage into the main memory to run it, execute the instructions in the program, and interact with the input and output devices to complete specific functions.

[0070] It should be noted that the concept of "server" used in this application can also be extended to the case of server clusters. Based on the network deployment principles understood by those skilled in the art, the servers should be logically divided. Physically, these servers can be independent of each other but accessible through interfaces, or they can be integrated into a single physical computer or a computer cluster. Those skilled in the art should understand this flexibility and should not use it to constrain the implementation of the network deployment method in this application.

[0071] One or more of the technical features of this application, unless explicitly specified herein, can be deployed on a server and accessed by a client remotely calling the online service interface provided by the server, or can be directly deployed and run on a client for access.

[0072] Unless otherwise specified, the neural network models referenced or potentially referenced in this application may be deployed on a remote server and invoked remotely on the client, or deployed on a client with the capability to invoke directly. In some embodiments, when running on the client, the corresponding intelligence may be acquired through transfer learning in order to reduce the requirements on the client's hardware resources and avoid excessive consumption of the client's hardware resources.

[0073] Unless otherwise specified, all data involved in this application may be stored remotely on a server or on a local terminal device, as long as it is suitable for use by the technical solution of this application.

[0074] Those skilled in the art will understand that although the various methods in this application are described based on the same concept and thus present commonality among them, they can be performed independently unless otherwise specified. Similarly, the various embodiments disclosed in this application are all based on the same inventive concept; therefore, concepts expressed in the same way, as well as concepts that are appropriately changed for convenience but are expressed differently, should be understood equivalently.

[0075] Unless otherwise expressly stated, the various embodiments disclosed in this application can be combined in a cross-cutting manner to flexibly construct new embodiments, as long as such combination does not depart from the inventive spirit of this application and can meet the needs of the prior art or solve a certain deficiency in the prior art. Those skilled in the art should be aware of such modifications.

[0076] Please see Figure 1 as well as Figure 2 In one embodiment of the image denoising method based on CLIP image encoder and Fourier structure encoder of this application, the method includes:

[0077] Step S10: Obtain the RGB image to be denoised containing real noise, so as to determine the semantic stream input data and structural stream input data corresponding to the RGB image to be denoised;

[0078] The image denoising system based on CLIP image encoder and Fourier structure encoder in the terminal device can acquire the RGB image to be denoised containing real noise in order to determine the semantic stream input data and structure stream input data corresponding to the RGB image to be denoised.

[0079] In some embodiments, the step of determining the semantic stream input data corresponding to the RGB image to be denoised includes:

[0080] Step S101: Obtain the RGB image to be denoised containing real noise, and perform normalization processing on the RGB image to be denoised in the same way as the CLIP image encoder pre-training stage.

[0081] Step S102: Perform linear pixel value transformation on the red, green and blue color channels of the RGB image to be denoised, respectively. Subtract the mean pixel value of the original pixel value of each color channel from the mean pixel value of that color channel, and then divide the mean-free pixel value by the standard deviation of the pixel value of that color channel to generate semantic stream input data that retains the original color and appearance information of the RGB image to be denoised.

[0082] Specifically, it receives a real-world RGB color image (to be denoised) containing camera sensor noise as the raw input, denoised as follows: Its dimensions are ,in, Indicates the image height. Indicates the image width. Corresponding to the red (R) channel, green (G) channel, and blue (B) channel.

[0083] For the RGB image to be denoised Perform a normalization process identical to that in the CLIP image encoder pre-training phase, which is a channel-independent linear transformation of pixel values. This is done on the RGB images to be denoised. Linear transformation of pixel values ​​is performed on the red, green, and blue color channels. The original pixel value of each channel is subtracted from the pixel mean of that color channel to eliminate the brightness shift between channels. The pixel value after removing the mean is divided by the pixel standard deviation of that color channel to unify the contrast scale between channels.

[0084] Through the above-described channel-wise linear transformation, semantic stream input data that preserves the original color and appearance information of the RGB image to be denoised is generated. This data will be used as the input of a parameter-frozen, pre-trained CLIP image encoder to extract a noise-robust global semantic feature map.

[0085] In some embodiments, the step of determining the structure stream input data corresponding to the RGB image to be denoised includes:

[0086] Step S1001: Obtain the RGB image to be denoised containing real noise, and separate the RGB image to be denoised into single-color channel images corresponding to the red color channel, green color channel, and blue color channel along the channel dimension;

[0087] Step S1002: Perform a two-dimensional discrete Fourier transform on each single color channel image to obtain its corresponding frequency domain representation, and decouple the phase spectrum representing the image structure layout from the frequency domain representation, while setting the amplitude spectrum representing the image intensity information in the frequency domain representation to a constant.

[0088] Step S1003: Perform an inversion operation on the decoupled phase spectrum, and then perform a two-dimensional discrete Fourier inverse transform on the inverted phase spectrum to obtain the phase inverted image corresponding to each single color channel image.

[0089] Step S1004: Reassemble the phase-inverted images corresponding to the red, green, and blue color channels along the channel dimension to generate structured stream input data that highlights the geometric structure, edges, and contour information of the image.

[0090] Specifically, it involves acquiring the RGB image to be denoised, which contains real noise. It is separated along the channel dimension into single-color channel images corresponding to the red, green, and blue color channels. The single-color channel image corresponding to the red channel is denoted as... The single-color channel image corresponding to the green color channel is denoted as The single-color channel image corresponding to the blue color channel is denoted as Subsequent frequency domain processing will be performed independently and in parallel on these three single-channel images.

[0091] For each single-channel image (collectively referred to as Perform a two-dimensional discrete Fourier transform to convert it from the spatial domain to the frequency domain, obtaining the complex frequency domain representation. This indicates that the real part is included. and the virtual part For each single-channel image (collectively referred to as...) The formula for calculating the two-dimensional discrete Fourier transform is expressed as follows:

[0092]

[0093] in, This represents a two-dimensional discrete Fourier transform operator that maps an image from the spatial domain to the frequency domain. Indicates the height of a single-channel image; Indicates the width of a single-channel image; For single-channel image components, it represents the noisy single-channel image corresponding to the red (R) channel, green (G) channel, and blue (B) channel; Represents the imaginary unit; The spatial coordinates of a single-channel image. Indicates the height dimension index. Indicates the width dimension index; For the frequency coordinates of a single-channel image, Indexed by the frequency domain height dimension. Indexed by the width dimension of the frequency domain.

[0094] Furthermore, the frequency domain representation Decoupling to amplitude spectrum and phase spectrum The amplitude spectrum By calculating the real part With the imaginary part The square root of the sum of squares is obtained, which reflects the intensity information of the image; wherein, the phase spectrum By calculating the imaginary part With the real part The arctangent is obtained, reflecting the structural layout of the image. The phase spectrum... The calculation formula is expressed as:

[0095]

[0096] in, This represents the phase spectrum extraction operator for a single-channel image, which outputs the phase spectrum of the single-channel image.

[0097] Frequency domain representation The imaginary part; Frequency domain representation The real part; This represents the arctangent function, used to calculate the argument of a complex number; For the frequency coordinates of a single-channel image, Indexed by the frequency domain height dimension. Indexed by the width dimension of the frequency domain.

[0098] To suppress intensity information that is closely coupled with noise, the amplitude spectrum is... Set to a constant or ignore the amplitude spectrum The influence of the phase spectrum, which characterizes the geometric structure of the image, is preserved. Used for subsequent calculations.

[0099] Furthermore, the phase spectrum obtained from decoupling... Perform a reversal operation to enhance high-frequency geometric features such as image edges and contours.

[0100] Furthermore, a two-dimensional discrete Fourier inverse transform is performed based on the inverted phase spectrum to map the signal back to the spatial domain, generating a phase-inverted image corresponding to each single-channel image. Subsequently, the phase-inverted images corresponding to the red, green, and blue channels are re-stitched along the channel dimensions to generate a dimension... Phase-reversed image .

[0101] Furthermore, the phase-reversed image As structural stream input data, it is fed into the subsequent learnable Fourier structure encoder. This data highlights the geometric structure, edges, and contour information of the image, while significantly suppressing noise interference, providing accurate structural guidance signals for the subsequent decoupling embedding module.

[0102] Step S20: Input the semantic stream input data into a parameter-frozen, pre-trained CLIP image encoder to extract a noise-robust global semantic feature map; input the structure stream input data into a learnable Fourier structure encoder isomorphic to the CLIP image encoder to extract a spatial structure feature map aligned with the spatial and channel dimensions of the global semantic feature map.

[0103] A denoised RGB image containing real noise is acquired. After determining the semantic stream input data and structure stream input data corresponding to the denoised RGB image, the semantic stream input data is input to a parameter-frozen, pre-trained CLIP image encoder to extract a noise-robust global semantic feature map. The structure stream input data is input to a learnable Fourier structure encoder isomorphic to the CLIP image encoder to extract a spatial structure feature map aligned with the spatial and channel dimensions of the global semantic feature map.

[0104] In this context, all weight parameters of the CLIP image encoder remain frozen throughout the feature extraction process and do not participate in gradient calculation or parameter updates; the number of network layers, number of channels, and downsampling strategy of the Fourier structure encoder are completely isomorphic to those of the CLIP image encoder.

[0105] In some embodiments, CLIP (Contrastive Language-Image Pre-Training) is a multimodal pre-trained model developed by OpenAI.

[0106] Specifically, it invokes the pre-trained CLIP image encoder with frozen parameters. As a core component of the semantic feature extraction branch, this encoder is pre-trained on massive image-text pairs, possessing inherent robustness to sensor distortions such as noise and blur. The semantic stream input data generated in step S102 is then input into the CLIP image encoder with its parameters frozen. The system outputs a series of multi-scale global semantic feature maps, denoted as... ,in, Represents the encoder level index (e.g.) These global semantic feature maps encode semantic information such as object categories and global context of the image, serving as stable global priors for the denoising task.

[0107] Furthermore, parallel calls to the learnable Fourier structure encoder As a core component of the structural feature extraction branch, this encoder adopts a network architecture that is completely isomorphic to the CLIP image encoder (including the same number of convolutional layers, number of channels, downsampling stride, and hierarchical structure) to ensure accurate alignment of the spatiotemporal and channel dimensions of subsequent feature fusion.

[0108] The structure stream generated in step S1004 above is input data (i.e., the phase inversion image). The input is fed into the learnable Fourier structure encoder, and the system outputs a series of spatial structure feature maps that are perfectly aligned with the spatial and channel dimensions of the semantic feature maps, denoted as... These structural feature maps are highly sensitive to high-frequency details and edge layouts of images, and are used to supplement the spatial details lost in the frozen semantic feature extraction branch.

[0109] Furthermore, to ensure effective subsequent fusion, the system strictly maintains the hierarchical correspondence between the two encoders during feature extraction. The CLIP encoder outputs a global semantic feature map in the first stage. Spatial structure feature map output from the first stage of the Fourier structure encoder One-to-one pairing, with both having completely identical spatial resolution and channel dimensions; until the fourth stage, four sets of semantic-structural feature pairs are finally formed. This hierarchical alignment mechanism ensures that subsequent decoupled embedding modules can perform pixel-level dynamic modulation at the same semantic and spatial scale, ultimately delivering the four pairs of dual-stream features to the next stage of the decoupled embedding module for processing.

[0110] After the above steps, this stage ultimately outputs four sets of semantic-structural feature pairs. Among them, the global semantic feature map... The semantic features in the image provide stable, noise-free content guidance, preventing artifacts or semantic changes in the denoised RGB image; spatial structure feature map The structural features within the map provide dynamic, detailed texture guidance for restoring image sharpness. Global semantic feature map Spatial structure feature diagram It will be simultaneously sent to the next stage of the decoupled embedding module for processing.

[0111] Step S30: For the global semantic feature map and spatial structure feature map of each level, take the spatial structure feature map as the generation condition, generate a channel dynamic filter based on global average pooling in the spatial dimension, generate a spatial dynamic filter by concatenating global average pooling and global max pooling in the channel dimension, concatenate the channel dynamic filter and the spatial dynamic filter and normalize to obtain a dynamic convolution kernel, and use the dynamic convolution kernel to adaptively convolve and modulate the global semantic feature map of the corresponding level to output a fused feature map;

[0112] The semantic stream input data is fed into a parameter-frozen, pre-trained CLIP image encoder to extract a noise-robust global semantic feature map. The structure stream input data is fed into a learnable Fourier structure encoder isomorphic to the CLIP image encoder to extract a spatial structure feature map aligned with the spatial and channel dimensions of the global semantic feature map. For each level of global semantic feature map and spatial structure feature map, a channel dynamic filter is generated based on global average pooling in the spatial dimension, using the spatial structure feature map as the generation condition. A spatial dynamic filter is generated by concatenating global average pooling and global max pooling in the channel dimension. The channel dynamic filter and the spatial dynamic filter are concatenated and normalized to obtain a dynamic convolution kernel. The dynamic convolution kernel is used to adaptively convolve and modulate the global semantic feature map of the corresponding level to output a fused feature map.

[0113] In some embodiments, for each level of global semantic feature map and spatial structure feature map, using the spatial structure feature map as a generation condition, a channel dynamic filter is generated based on global average pooling in the spatial dimension. A spatial dynamic filter is generated by concatenating global average pooling and global max pooling in the channel dimension. The channel dynamic filter and the spatial dynamic filter are concatenated and normalized to obtain a dynamic convolution kernel. The dynamic convolution kernel is then used to adaptively convolve and modulate the global semantic feature map of the corresponding level to output a fused feature map. The steps include:

[0114] Step S301: For each level of global semantic feature map and spatial structure feature map obtained after feature extraction of the RGB image to be denoised, the spatial structure feature map is used as the generation condition to perform global average pooling of the spatial dimension to generate a channel dynamic filter.

[0115] Specifically, please refer to Figure 3 In the Decoupled Embedding Module (DEM), for each level obtained after feature extraction from the RGB image to be denoised, a global semantic feature map from the semantic feature extraction branch is simultaneously received. Spatial structural feature map of structural feature extraction branch The global semantic feature map and the spatial structure feature map Having the same spatial resolution (height) ,width ) and number of channels This ensures that the two are strictly aligned in space.

[0116] Based on the aforementioned spatial structure feature map To generate the conditions, global average pooling is performed on the spatial dimension, compressing the spatial information into statistical descriptors of the channel dimension, thus generating channel dynamic filters. Its specific implementation includes:

[0117] Spatial structure feature map Local windows (e.g.) The scope performs global average pooling (GAP) on the spatial dimension, compressing the spatial information into... Feature descriptors .

[0118] Then, the feature descriptor Input to a convolution kernel of size The convolutional layers undergo a linear transformation, mapping the channel dimension to the size of the convolutional kernel. The dynamic filter for this channel The importance weights of the current region on different channels are encoded, and the process can be represented as follows:

[0119]

[0120] in, This represents a channel dynamic filter that encodes the importance weights of different channels in the current region. This represents a linear mapping operator, such as a convolution kernel size of . Linear transformation of the convolutional layer; Represents the spatial dimension global average pooling (GAP) operator; Representation of spatial structure feature map The In the hierarchy, based on spatial pixel position The local spatial structure feature block within the K×K neighborhood centered on the element; Representation of spatial structure feature map Spatial pixel position .

[0121] For spatial structure feature maps Each spatial pixel position Utilizing its corresponding structural features Generate unique channel dynamic filters to modulate the corresponding semantic features. .

[0122] Step S302: Simultaneously perform global average pooling and global max pooling on the spatial structure feature map along the channel dimension to obtain a first pooling result and a second pooling result. Then, concatenate the first pooling result and the second pooling result along the channel dimension to generate a spatial dynamic filter.

[0123] The system obtains spatial structure feature maps in parallel. Extracting spatial dimension structural information is used to generate spatial dynamic filters. Spatial structure feature map exist Within the local spatial neighborhood, global average pooling and global max pooling are performed along the channel dimension, respectively. Global average pooling is used to capture background structure information, yielding a first pooling result; global max pooling is used to capture significant edge texture information, yielding a second pooling result. The first and second pooling results are concatenated along the channel dimension to generate a spatial dynamic filter that encodes the spatial structure layout within the local neighborhood. The process can be represented as follows:

[0124]

[0125] in, This represents a spatial dynamic filter used to encode the spatial structure layout within a local neighborhood. This represents a combination of channel-level global average pooling, channel-level global max pooling, and channel-level concatenation operations. express Representation of spatial structure feature map The In the hierarchy, based on spatial pixel position The local spatial structure feature block within the K×K neighborhood centered on the locator.

[0126] Step S303: Concatenate the channel dynamic filter and the spatial dynamic filter along the channel dimension to obtain the original dynamic convolution kernel, and normalize the original dynamic convolution kernel to obtain the dynamic convolution kernel.

[0127] The channel dynamic filter generated in step S301 The spatial dynamic filter generated in step S302 By concatenating along the channel dimension, we obtain the original dynamic convolution kernel containing complete spatiotemporal information.

[0128] Since the predicted filter parameters directly depend on the input content, their values ​​may fluctuate significantly. To prevent gradient explosion or vanishing during training, the system performs a normalization operation on the original dynamic convolution kernel. By constraining the numerical values ​​within a reasonable range, the final dynamic convolution kernel is obtained. The process can be represented as follows:

[0129]

[0130] in, This represents the dynamic convolution kernel after normalization. This indicates a normalization operation; This indicates a splicing operation along the channel dimension.

[0131] Step S304: The pixel regions of the global semantic feature map at the corresponding level are subjected to element-wise weighted summation and feature modulation by the dynamic convolution kernel, and the spatial structure features are embedded into the global semantic features to output a fused feature map that integrates global semantic information and spatial structure details.

[0132] Utilizing the generated dynamic convolution kernel For the global semantic feature map of the corresponding level Element-wise weighted summation and feature modulation are performed on the pixel region to embed spatial structure features into global semantic features.

[0133] For global semantic feature map Each center pixel on Utilizing its corresponding dynamic convolution kernel and its surroundings Element-wise broadcast multiplication and weighted summation are performed on semantic pixels within the neighborhood to achieve adaptive feature embedding. This step selectively enhances or suppresses feature responses in the semantic flow through weights provided by the structure flow. Its calculation formula can be expressed as:

[0134]

[0135] in, Indicates the first Channel, pixel position ( , Modulated characteristic value at ) This represents the summation operation over all offset positions within the convolution window; Indicates the size of the convolution kernel (e.g., 1×1, 3×3, etc.). Represents the offset coordinates within the window relative to the center pixel; Indicates the first Channel, pixel position ( , At this location, offset ( , The corresponding dynamic convolution kernel weights; Represents the set of offsets for the convolution window; This represents the Hadamard product (element-by-element multiplication) operator; Indicates the first level, number Channel, offset position ( , Global semantic features at ().

[0136] Features that have undergone dynamic modulation Aggregation (e.g., through summation) is performed to generate a fused feature map that combines global semantic information with spatial structural details. This feature retains both the distortion-invariant semantics of the semantic feature extraction branch and the fine structural texture of the structural feature extraction branch, serving as high-quality input for subsequent processing.

[0137] Step S40: The fused feature map is uniformly divided into multiple sub-feature groups along the channel dimension. For each sub-feature group, all other sub-feature groups are concatenated along the channel dimension to form complementary context features. The affine transformation module predicts affine parameters from the complementary context features to perform affine modulation on the current sub-feature group. All affine-modulated sub-feature groups are recombined and smoothed by a convolutional layer with a kernel size of 3×3 to output a refined feature map.

[0138] For each level of global semantic feature map and spatial structure feature map, a channel dynamic filter is generated based on the spatial structure feature map as the generation condition, using global average pooling in the spatial dimension. A spatial dynamic filter is generated by concatenating global average pooling and global max pooling in the channel dimension. The channel dynamic filter and the spatial dynamic filter are concatenated and normalized to obtain a dynamic convolution kernel. The dynamic convolution kernel is used to adaptively convolve and modulate the global semantic feature map of the corresponding level to output a fused feature map. The fused feature map is then uniformly divided into multiple sub-feature groups along the channel dimension. For each sub-feature group, all other sub-feature groups are concatenated along the channel dimension to form complementary context features. The affine transformation module (ATM) predicts affine parameters from the complementary context features to perform affine modulation on the current sub-feature group. All affine-modulated sub-feature groups are recombined and smoothed by a convolutional layer with a kernel size of 3×3 to output a refined feature map.

[0139] After the features are initially fused by the Decoupled Embedding Module (DEM), they enter the feature refinement and interaction stage. To address the lack of deep dependency modeling between feature channels, an inter-channel interaction module is introduced in this stage. This module uses a "segmentation-interaction-recombination" strategy to perform adaptive calibration by utilizing complementary information between channels.

[0140] In some embodiments, the fused feature map is uniformly divided into multiple sub-feature groups along the channel dimension. For each sub-feature group, all other sub-feature groups are concatenated along the channel dimension to form complementary context features. Affine parameters are predicted from the complementary context features using an affine transformation module to perform affine modulation on the current sub-feature group. All affine-modulated sub-feature groups are recombined and smoothed through a 3×3 convolutional layer to output a refined feature map. The steps include:

[0141] Step S401: The fused feature map, which integrates global semantic information and spatial structural details, is uniformly divided along the channel dimension to obtain multiple sub-feature groups with consistent channel dimensions;

[0142] The system receives a fused feature map from the previous stage, which integrates global semantic information and spatial structural details, denoted as... .

[0143] Please see Figure 4 In the Interchannel Interaction (MCI) module, to reduce computational complexity and achieve group interaction, the fused feature map is first uniformly divided along the channel dimension to obtain... Each sub-feature group has the same channel dimension. In this embodiment, the number of segments... The value can be set to 4 to balance computational efficiency and performance. After segmentation, sub-feature group sequences are obtained. Each sub-feature group The dimension is , , , These represent the height, width, and total number of channels of the fused feature map, respectively.

[0144] Step S402: For each sub-feature group, concatenate all other sub-feature groups except the current sub-feature group along the channel dimension to generate complementary context features corresponding to the current sub-feature group.

[0145] For each sub-feature group ,in , except for the current sub-feature group All other sub-feature groups except the current one are concatenated along the channel dimension to generate complementary context features corresponding to the current sub-feature group. .

[0146] Complementary context features Includes the rest of the feature stream Global information for each part, with dimensions of This step ensures that subsequent calibrations are performed based on the global context.

[0147] Step S403: Input the complementary context features into the affine transformation module, and the affine transformation module predicts the affine parameters for the current sub-feature group. Then, use the affine parameters to perform affine modulation processing on the current sub-feature group.

[0148] The complementary context features The input is fed into an affine transformation module (ATM), which predicts a scaling factor for the current sub-feature group. and offset factor The affine parameters; using the scaling factor and offset factor For the current sub-feature group Affine modulation processing is performed; wherein, the affine transform module (ATM) contains a lightweight convolutional neural network structure, which sequentially includes a first convolutional layer with a kernel size of 1×1, a ReLU activation function, and a second convolutional layer with a kernel size of 1×1, outputting two sets of features related to the current sub-feature group. Dimension matching includes scaling factor and offset factor The affine parameters are used to learn the distribution state that the current channel should adjust from the complementary context.

[0149] Furthermore, a scaling factor is used. and offset factor The affine parameters for the current sub-feature group The formula for performing a linear affine transformation is as follows:

[0150]

[0151] in, Indicates the first Features of a sub-feature group after affine modulation; Indicates the first The scaling factor of each sub-feature group is predicted by the affine transformation module; Indicates the first Sub-feature groups; Indicates the first The offset factors of each sub-feature group are predicted by the affine transformation module. ⊙ represents the Hadamard product (element-wise multiplication) operator, corresponding to the element-wise multiplication of pixels at the same position in the feature map.

[0152] By scaling factor Adjust the activation intensity of the feature using the offset factor. Adjusting the baseline value of the feature allows for the calibration of the current channel representation using information from other channels, thus enabling inter-channel interaction.

[0153] Step S404: Reassemble all the affine-modulated sub-feature groups along the channel dimension, input the reassembled feature map into a convolutional layer with a kernel size of 3×3 for spatial information smoothing, and output a refined feature map with channel interaction refinement.

[0154] In all After each sub-feature group completes the processing steps S403 to S404 in parallel, the system combines all calibrated sub-feature groups. By re-cascading along the channel dimensions, a reconstructed feature map is obtained, whose dimensions are restored to [value missing]. .

[0155] To eliminate the boundary effects caused by channel segmentation and further fuse spatial information, the reconstructed feature map is input into a 3×3 convolutional layer for fusion. The final output is a refined feature map. This refined feature map It has stronger cross-channel dependency and structural sensitivity, and is then transmitted to the next level of the decoder or to the reconstruction module via a skip connection.

[0156] Step S50: Call the preset decoder to sequentially perform feature upsampling, symmetric skip connection fusion, and residual convolution refinement on the refined feature map, restoring it layer by layer to the original image resolution of the RGB image to be denoised, obtaining high-dimensional feature data that matches the original image resolution, and mapping the high-dimensional feature data to the denoised RGB image through a convolutional layer with a kernel size of 3×3.

[0157] The fused feature map is uniformly divided into multiple sub-feature groups along the channel dimension. For each sub-feature group, all other sub-feature groups are concatenated along the channel dimension to form complementary context features. The affine parameters are predicted from the complementary context features by the affine transformation module to perform affine modulation on the current sub-feature group. All affine-modulated sub-feature groups are recombined and smoothed by a convolutional layer with a kernel size of 3×3. After outputting a refined feature map, a preset decoder is called to sequentially perform feature upsampling, symmetric skip connection fusion, and residual convolution refinement on the refined feature map, restoring it layer by layer to the original image resolution of the RGB image to be denoised, obtaining high-dimensional feature data matching the original image resolution. The high-dimensional feature data is then mapped to the denoised RGB image by a convolutional layer with a kernel size of 3×3.

[0158] After feature extraction, fusion, and refinement are completed, the image reconstruction and output stage begins. This stage utilizes a hierarchical decoder to gradually map the deep, high-dimensional features back to the pixel space of the original image.

[0159] In some embodiments, a preset decoder is invoked to sequentially perform feature upsampling, symmetric skip connection fusion, and residual convolution refinement on the refined feature map, restoring it layer by layer to the original image resolution of the RGB image to be denoised, thereby obtaining high-dimensional feature data matching the original image resolution. The step of mapping the high-dimensional feature data to the denoised RGB image through a convolutional layer with a kernel size of 3×3 includes:

[0160] Step S501: Construct a decoder that is structurally symmetrical with the CLIP image encoder and the Fourier structure encoder, and input the refined feature map into the decoder;

[0161] The system constructs a decoder whose structure is symmetrical to that of the CLIP image encoder and the Fourier structure encoder.

[0162] The decoder contains the same number of layers as the encoder (e.g., 4 stages). The decoding process is a bottom-up process, starting from the deepest bottleneck of the network and gradually restoring the spatial resolution of the original image.

[0163] Step S502: For the refined feature map, feature upsampling, symmetric skip connection fusion, and residual convolution refinement are performed sequentially at each level of the decoder to gradually restore the spatial resolution of the feature map to the original image resolution of the RGB image to be denoised, thereby obtaining high-dimensional feature data that matches the original image resolution.

[0164] The refined feature map is processed sequentially at each level of the decoder as follows: upsampling, skip connection fusion, residual convolution refinement, etc. The feature upsampling, for the first level of the decoder, receives data from deeper levels (the...). The refined feature map of the encoder layer is then subjected to an upsampling operation (such as transposed convolution or bilinear interpolation) to double its spatial resolution, thereby aligning it with the feature size of the corresponding layer in the encoder. Matching. The symmetric skip connection fusion means that, while upsampling, fused features from the encoder are introduced through symmetric skip connections. This fusion feature It contains CLIP semantics and Fourier structure information; the upsampled decoder features are then fused with this feature. Concatenation along the channel dimension forms a combined feature containing rich multi-scale information, mitigating gradient vanishing and supplementing reconstruction details. The residual convolutional refinement involves inputting the concatenated combined feature into a residual convolutional block, utilizing stacked convolutional layers and residual connections for nonlinear transformation and refinement, smoothing upsampling artifacts and restoring pixel details; the processed features are then transmitted to a shallower layer of the decoder. (Layer); repeat step S502 until the spatial resolution is restored to the original resolution of the RGB image to be denoised, and obtain high-dimensional feature data that matches the resolution of the original image.

[0165] Step S503: Input the high-dimensional feature data into a convolutional layer with a kernel size of 3×3, and perform feature dimension mapping data processing through the convolutional layer to map the high-dimensional feature data into a denoised RGB image.

[0166] Once the decoder has completed processing at all levels, it restores the image to its original resolution. Then, a terminal convolutional layer (typically a 3×3 kernel layer) maps the high-dimensional feature data into 3-channel RGB image data. The network can then directly predict the denoised RGB image. Alternatively, the network can predict the noise residual map and add it to the denoised RGB image containing real noise to obtain the denoised RGB image.

[0167] In some embodiments, during the network training phase, the system needs to calculate the difference between the predicted results and the true values ​​to optimize the learnable parameters. (Acquiring and inputting noisy images) The corresponding real noise-free image is denoted as The mean squared error (MSE / L2Loss) is used as the objective function, and its calculation formula is expressed as follows:

[0168]

[0169] in, Represented by learnable parameters The mean squared error loss value is used to measure the difference between the denoised RGB image and the RGB image to be denoised. Indicates the number of samples in a batch during training; This represents the overall mapping relationship of the CLIP-guided adaptive Fourier embedding network; Indicates the first An RGB image containing real noise to be denoised Indicates the first A real, noise-free RGB image (label image); This represents the square of the L2 norm, i.e., the square of the Euclidean norm, used to calculate the sum of squared errors between pixels.

[0170] This represents the set of learnable parameters in the network (excluding frozen CLIP encoder parameters).

[0171] Based on the calculated loss value The backpropagation algorithm and Adam optimizer are used to update the weight parameters of the Fourier encoder, decoupled embedding module, interchannel interaction scheme, and decoder. Special note: During this process, the parameters of the CLIP image encoder... It will remain frozen and will not be updated.

[0172] As can be seen from the above embodiments, compared with the prior art, the present application addresses the problems of insufficient generalization ability, difficulty in balancing efficiency and performance, and incomplete feature representation in existing real image denoising techniques. The present application has, but is not limited to, the following beneficial effects:

[0173] Firstly, this application introduces a parameter-frozen pre-trained CLIP image encoder to extract global semantic features. The distortion invariance obtained by CLIP during training on massive image and text data makes its extracted features naturally robust to sensor distortions such as noise and blur, allowing the model to avoid overfitting to specific training noise distributions. Simultaneously, combined with Fourier embedding for structural feature extraction, it can adapt to complex real-world noise generated by unknown shooting devices and extreme shooting environments without needing to accurately model the complex distribution of real-world noise. This effectively solves the "domain difference" problem of traditional methods and significantly improves the model's cross-domain generalization performance in real-world scenarios.

[0174] Secondly, this application achieves the separation, extraction, and precise fusion of semantic priors and structural details through a dual-stream architecture: the CLIP image encoder provides a stable global semantic prior, ensuring the semantic consistency of the denoising results and avoiding problems such as artifacts and semantic distortion; the Fourier structure encoder extracts spatial structural features from the structural flow data that enhances geometric structures, accurately capturing high-frequency details such as edges and textures that are submerged by noise. Simultaneously, by using dynamic convolutional kernels, structural information is finely embedded into semantic features, achieving adaptive modulation of semantic priors by structural details. This allows the denoising results to effectively remove noise while maximizing the recovery of sharp edges and rich textures in the image, solving the problems of incomplete single-domain feature representation and semantic-detail imbalance in traditional methods.

[0175] Thirdly, the adaptive mechanism of this application adopts a lightweight design, effectively solving the problems of high computational cost and low parameter efficiency in traditional dynamic filtering methods. On the one hand, it dynamically generates lightweight convolution kernels based on structural features, performing pixel-level adaptive modulation on semantic features, thus avoiding the huge parameter overhead of traditional dynamic convolution. On the other hand, it achieves inter-channel interaction by using channel grouping and lightweight affine transformation, completing feature refinement only through simple 1×1 convolution, element-wise operations, and 3×3 convolution smoothing, resulting in extremely low computational cost. While maintaining top-tier denoising performance, the overall model has a small model size and fast inference speed, meeting the power consumption and real-time requirements of resource-constrained terminal devices such as smartphones, drones, and surveillance cameras, making it easier to implement in practical applications.

[0176] Please see Figure 5An image denoising device based on CLIP image encoder and Fourier structure encoder is provided to meet one of the purposes of this application, including a data acquisition module 1100, a feature map extraction module 1200, a fusion feature map construction module 1300, a refined feature map construction module 1400, and an image denoising module 1500. The data acquisition module 1100 is configured to acquire an RGB image to be denoised containing real noise, in order to determine the semantic stream input data and structure stream input data corresponding to the RGB image to be denoised; the feature map extraction module 1200 is configured to input the semantic stream input data into a parameter-frozen, pre-trained CLIP image encoder to extract a global semantic feature map robust to noise; and input the structure stream input data into a learnable Fourier structure encoder isomorphic to the CLIP image encoder to extract a spatial structure feature map aligned with the spatial and channel dimensions of the global semantic feature map; the feature map fusion construction module 1300 is configured to, for each level of global semantic feature map and spatial structure feature map, generate a channel dynamic filter based on global average pooling in the spatial dimension using the spatial structure feature map as the generation condition, generate a spatial dynamic filter by concatenating global average pooling and global max pooling in the channel dimension, and concatenate the channel dynamic filter with the spatial dynamic filter and normalize it to obtain... A dynamic convolutional kernel is used to adaptively convolve and modulate the global semantic feature map of the corresponding layer to output a fused feature map. A refined feature map construction module 1400 is configured to uniformly divide the fused feature map into multiple sub-feature groups along the channel dimension. For each sub-feature group, all other sub-feature groups are concatenated along the channel dimension to form complementary context features. The affine transformation module predicts affine parameters from the complementary context features to perform affine modulation on the current sub-feature group. All affine-modulated sub-feature groups are recombined and smoothed by a convolutional layer with a kernel size of 3×3 to output a refined feature map. An image denoising module 1500 is configured to call a preset decoder to sequentially perform feature upsampling, symmetric skip connection fusion, and residual convolution refinement on the refined feature map, restoring it layer by layer to the original image resolution of the RGB image to be denoised, obtaining high-dimensional feature data matching the original image resolution. The high-dimensional feature data is mapped to the denoised RGB image by a convolutional layer with a kernel size of 3×3.

[0177] Based on any embodiment of this application, please refer to Figure 6 Another embodiment of this application also provides an electronic device, which can be implemented by a computer device, such as... Figure 6The diagram shows the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable storage medium stores an operating system, a database, and computer-readable instructions. The database stores control information sequences. When executed by the processor, the computer-readable instructions enable the processor to implement an image denoising method based on a CLIP image encoder and a Fourier structure encoder. The processor provides computational and control capabilities, supporting the operation of the entire computer device. The memory stores computer-readable instructions, which, when executed by the processor, enable the processor to execute the image denoising method based on a CLIP image encoder and a Fourier structure encoder as described in this application. The network interface of the computer device is used for communication with a terminal. Those skilled in the art will understand that… Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0178] In this embodiment, the processor is used to execute... Figure 5 The memory stores the specific functions of each module, and stores the program code and various data required to execute the above modules. The network interface is used for data transmission between the user terminal and the server. In this embodiment, the memory stores the program code and data required to execute all modules in the image denoising device based on CLIP image encoder and Fourier structure encoder of this application. The server can call the server's program code and data to execute the functions of all modules.

[0179] This application also provides a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the image denoising method based on CLIP image encoder and Fourier structure encoder described in any embodiment of this application.

[0180] This application also provides a computer program product, including a computer program / instructions that, when executed by one or more processors, implement the steps of the image denoising method based on CLIP image encoder and Fourier structure encoder described in any embodiment of this application.

[0181] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0182] The above description is only a partial embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An image denoising method based on CLIP image encoder and Fourier structure encoder, characterized in that, include: Obtain the RGB image to be denoised, which contains real noise, to determine the semantic stream input data and structural stream input data corresponding to the RGB image to be denoised; The semantic stream input data is input to a parameter-frozen, pre-trained CLIP image encoder to extract a noise-robust global semantic feature map; the structure stream input data is input to a learnable Fourier structure encoder isomorphic to the CLIP image encoder to extract a spatial structure feature map aligned with the spatial and channel dimensions of the global semantic feature map. For each level of global semantic feature map and spatial structure feature map, the spatial structure feature map is used as the generation condition. A channel dynamic filter is generated based on global average pooling in the spatial dimension. A spatial dynamic filter is generated by concatenating global average pooling and global max pooling in the channel dimension. The channel dynamic filter and the spatial dynamic filter are concatenated and normalized to obtain a dynamic convolution kernel. The dynamic convolution kernel is used to adaptively convolve and modulate the global semantic feature map of the corresponding level to output a fused feature map. The fused feature map is uniformly divided into multiple sub-feature groups along the channel dimension. For each sub-feature group, all other sub-feature groups are concatenated along the channel dimension to form complementary context features. The affine transformation module predicts affine parameters from the complementary context features to perform affine modulation on the current sub-feature group. All affine-modulated sub-feature groups are recombined and smoothed by a convolutional layer with a kernel size of 3×3 to output a refined feature map. The preset decoder is invoked to sequentially perform feature upsampling, symmetric skip connection fusion, and residual convolution refinement on the refined feature map, restoring it layer by layer to the original image resolution of the RGB image to be denoised, thereby obtaining high-dimensional feature data that matches the original image resolution. The high-dimensional feature data is then mapped to the denoised RGB image through a convolutional layer with a kernel size of 3×3.

2. The image denoising method based on CLIP image encoder and Fourier structure encoder according to claim 1, characterized in that, The step of determining the semantic stream input data corresponding to the RGB image to be denoised includes: Obtain the RGB image to be denoised, which contains real noise, and perform normalization processing on the RGB image to be denoised in the same way as in the CLIP image encoder pre-training stage. Linear pixel value transformation is performed on the red, green, and blue color channels of the RGB image to be denoised. The original pixel value of each color channel is subtracted from the mean pixel value of that color channel, and then the mean-free pixel value is divided by the standard deviation of the pixel value of that color channel to generate semantic stream input data that retains the original color and appearance information of the RGB image to be denoised.

3. The image denoising method based on CLIP image encoder and Fourier structure encoder according to claim 1, characterized in that, The step of determining the structured stream input data corresponding to the RGB image to be denoised includes: Obtain the RGB image to be denoised, which contains real noise, and separate the RGB image to be denoised into single-color channel images corresponding to the red, green, and blue color channels along the channel dimension; A two-dimensional discrete Fourier transform is performed on each single-color channel image to obtain its corresponding frequency domain representation. The phase spectrum representing the image structure layout is decoupled from the frequency domain representation, and the amplitude spectrum representing the image intensity information in the frequency domain representation is set to a constant. The phase spectrum obtained by decoupling is inverted, and then the inverted phase spectrum is subjected to a two-dimensional discrete Fourier inverse transform to obtain the phase inverted image corresponding to each single color channel image. The phase-reversed images corresponding to the red, green, and blue color channels are re-stitched along the channel dimensions to generate structured stream input data that highlights the geometric structure, edges, and contour information of the image.

4. The image denoising method based on CLIP image encoder and Fourier structure encoder according to claim 1, characterized in that, For each level of global semantic feature map and spatial structure feature map, using the spatial structure feature map as the generation condition, a channel dynamic filter is generated based on global average pooling in the spatial dimension. A spatial dynamic filter is generated by concatenating global average pooling and global max pooling in the channel dimension. The channel dynamic filter and the spatial dynamic filter are concatenated and normalized to obtain a dynamic convolution kernel. The dynamic convolution kernel is then used to adaptively convolve and modulate the global semantic feature map of the corresponding level to output a fused feature map. The steps include: For each level of global semantic feature map and spatial structure feature map obtained after feature extraction of the RGB image to be denoised, the spatial structure feature map is used as the generation condition to perform global average pooling of the spatial dimension to generate a channel dynamic filter. The spatial structure feature map is simultaneously subjected to global average pooling and global max pooling in the channel dimension to obtain a first pooling result and a second pooling result. The first pooling result and the second pooling result are then concatenated along the channel dimension to generate a spatial dynamic filter. The channel dynamic filter and the spatial dynamic filter are concatenated along the channel dimension to obtain the original dynamic convolution kernel. The original dynamic convolution kernel is then normalized to obtain the dynamic convolution kernel. The dynamic convolution kernel performs element-wise weighted summation and feature modulation on the pixel regions of the global semantic feature map at the corresponding level, embedding spatial structure features into the global semantic features, and outputting a fused feature map that integrates global semantic information and spatial structure details.

5. The image denoising method based on CLIP image encoder and Fourier structure encoder according to claim 1, characterized in that, The steps include: uniformly dividing the fused feature map into multiple sub-feature groups along the channel dimension; for each sub-feature group, concatenating all other sub-feature groups along the channel dimension to form complementary context features; predicting affine parameters from the complementary context features using an affine transformation module to perform affine modulation on the current sub-feature group; recombining all affine-modulated sub-feature groups; smoothing the results through a 3×3 convolutional layer; and outputting a refined feature map. The fused feature map, which integrates global semantic information and spatial structural details, is uniformly divided along the channel dimension to obtain multiple sub-feature groups with consistent channel dimensions. For each sub-feature group, all other sub-feature groups except the current sub-feature group are concatenated along the channel dimension to generate complementary context features corresponding to the current sub-feature group; The complementary context features are input into the affine transformation module, which predicts the affine parameters for the current sub-feature group. The affine parameters are then used to perform affine modulation processing on the current sub-feature group. All affine-modulated sub-feature groups are reassembled along the channel dimension. The reassembled feature map is then input into a 3×3 convolutional layer for spatial smoothing, and the refined feature map with channel interaction refinement is output.

6. The image denoising method based on CLIP image encoder and Fourier structure encoder according to claim 1, characterized in that, The process involves calling a preset decoder to sequentially perform feature upsampling, symmetric skip connection fusion, and residual convolution refinement on the refined feature map, layer by layer restoring it to the original image resolution of the RGB image to be denoised, thereby obtaining high-dimensional feature data matching the original image resolution. The high-dimensional feature data is then mapped to the denoised RGB image through a 3×3 convolutional layer. This process includes: Construct a decoder that is structurally symmetrical to the CLIP image encoder and the Fourier structure encoder, and input the refined feature map into the decoder; The refined feature map is sequentially subjected to feature upsampling, symmetric skip connection fusion, and residual convolution refinement at each level of the decoder, gradually restoring the spatial resolution of the feature map to the original image resolution of the RGB image to be denoised, thereby obtaining high-dimensional feature data that matches the original image resolution; The high-dimensional feature data is input into a convolutional layer with a kernel size of 3×3. The convolutional layer performs feature dimension mapping data processing to map the high-dimensional feature data into a denoised RGB image.

7. The image denoising method based on a CLIP image encoder and a Fourier structure encoder according to any one of claims 1 to 6, characterized in that, All weight parameters of the CLIP image encoder remain frozen throughout the feature extraction process and do not participate in gradient calculation and parameter updates. The number of network layers, number of channels, and downsampling strategy of the Fourier structure encoder are completely isomorphic to those of the CLIP image encoder.

8. An image denoising device based on a CLIP image encoder and a Fourier structure encoder, characterized in that, include: The data acquisition module is configured to acquire the RGB image to be denoised, which contains real noise, in order to determine the semantic stream input data and structural stream input data corresponding to the RGB image to be denoised. The feature map extraction module is configured to input the semantic stream input data into a parameter-frozen, pre-trained CLIP image encoder to extract a noise-robust global semantic feature map; and input the structure stream input data into a learnable Fourier structure encoder isomorphic to the CLIP image encoder to extract a spatial structure feature map aligned with the spatial and channel dimensions of the global semantic feature map. The fusion feature map construction module is configured to generate channel dynamic filters based on global average pooling in the spatial dimension, using the spatial structure feature map as the generation condition, and then generate spatial dynamic filters by concatenating global average pooling and global max pooling in the channel dimension. The channel dynamic filters and the spatial dynamic filters are then concatenated and normalized to obtain dynamic convolution kernels. The dynamic convolution kernels are used to adaptively convolve and modulate the global semantic feature maps of the corresponding layers to output fusion feature maps. The refined feature map construction module is configured to uniformly divide the fused feature map into multiple sub-feature groups along the channel dimension. For each sub-feature group, all other sub-feature groups are concatenated along the channel dimension to form complementary context features. The affine transformation module predicts affine parameters from the complementary context features to perform affine modulation on the current sub-feature group. All affine-modulated sub-feature groups are recombined and smoothed by a convolutional layer with a kernel size of 3×3 to output the refined feature map. The image denoising module is configured to call a preset decoder to sequentially perform feature upsampling, symmetric skip connection fusion, and residual convolution refinement on the refined feature map, restoring it layer by layer to the original image resolution of the RGB image to be denoised, thereby obtaining high-dimensional feature data that matches the original image resolution. The high-dimensional feature data is then mapped to the denoised RGB image through a convolutional layer with a kernel size of 3×3.

9. An electronic device comprising a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implemented according to any one of claims 1 to 7, which, when invoked by a computer, executes the steps included in the corresponding method.

Images