Integrated image restoration method based on frequency modulation and structure prior

Abstract

The invention particularly relates to an integrated image restoration method based on frequency modulation and structure prior. The method comprises the following steps: performing convolution operation on an input degraded image to extract shallow layer features; inputting the shallow features into an encoder for processing, and extracting structure and context information under different scales to obtain multi-scale feature representation; structure prior is introduced into jump connection, a gating mechanism is combined, jump connection features from an encoder are processed, and modulated output features are obtained; processing the trunk features output by the decoder through a high and low frequency decoupling network, and decomposing the trunk features into low frequency features and high frequency features; performing enhancement, bidirectional interaction and frequency domain modulation on the low-frequency features and the high-frequency features by using a bidirectional interaction modulation network to obtain fusion features; and the decoder maps the fusion feature into a high-quality restored image consistent with the input size by adopting a layer-by-layer up-sampling and convolution reconstruction strategy. According to the method, the image restoration definition and the structure consistency are improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to an integrated image restoration method based on frequency modulation and structural prior. Background Technology

[0002] Image restoration is a crucial task in computer vision, aiming to reconstruct clear images from degraded or low-quality observations. This not only improves visual quality but also supports applications such as autonomous navigation, surveillance, and medical imaging. Achieving high-quality, robust image restoration remains a significant challenge due to the frequent impact of coupled degradation factors such as noise, blur, haze, and low resolution during the observation and imaging process. Traditional methods, often based on task-specific modeling, rely on manual priors or fixed degradation assumptions, achieving good results only in single degradation scenarios and failing to address coupled degradation problems in complex real-world environments. In recent years, integrated image restoration has aimed to address multiple degradations simultaneously using a single model, thereby improving its versatility and adaptability. These methods typically focus on task awareness and conditional guidance, achieving joint restoration of multiple degradations through degradation feature recognition, feature separation, and information modulation. This approach effectively expands the research paradigm of integrated image restoration and demonstrates good application potential in complex degradation scenarios. However, existing methods largely concentrate on spatial domain modeling, lacking sufficient structural consistency and detail fidelity. Frequency domain modeling provides an effective approach for image restoration, characterizing global contours at the low-frequency level and recovering texture and edge information at the high-frequency level, thus offering a more comprehensive feature representation for image restoration. However, most existing frequency domain methods remain at the level of overall frequency distribution or channel-level fusion, lacking in-depth modeling of the complementary relationship between low and high frequencies, and therefore their effectiveness is still limited in complex degradation scenarios. Therefore, how to efficiently utilize frequency domain information and achieve adaptive fusion between low and high frequencies remains a key challenge for integrated image restoration.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] To overcome the shortcomings of existing integrated image restoration methods based on deep learning, this invention provides an integrated image restoration method based on frequency modulation and structural prior.

[0005] Other features and advantages of the invention will become apparent from the following detailed description, or may be learned in part by practice of the invention.

[0006] According to a first aspect of the present invention, an integrated image restoration method based on frequency modulation and structural prior is provided, the method comprising: Step 1: Perform a 3×3 convolution operation on the input degraded image to extract shallow features. ; Step 2: Shallow features The input is processed in the encoder to extract structural and contextual information at different scales, resulting in multi-scale feature representations. ; Step 3: Introduce structural priors and combine them with gating mechanisms in skip connections to process features from multiple scales. Skip connection features obtained after each Transformer block in the input encoder The modulated output characteristics are obtained. ; Step 4: After encoder processing, the signal enters the decoder stage. A high-low frequency decoupling network is used to process the backbone features output by each Transformer block in the decoder. The process involves decomposing the data into low-frequency features. High frequency characteristics ; Step 5: Utilize a bidirectional interactive modulation network to analyze low-frequency features. High frequency characteristics Enhancement, bidirectional interaction, and frequency domain modulation are performed to obtain fusion features; Step 6: The decoder uses a layer-by-layer upsampling and convolutional reconstruction strategy to map the fused features into a high-quality restored image with the same size as the input.

[0007] In some exemplary embodiments, the encoder in step 2 includes four stages, with adjacent stages achieving a halving of resolution and a doubling of the number of channels through convolutional downsampling operations.

[0008] In some exemplary embodiments, step 3 specifically includes the following sub-steps: Sub-step 3-1: Apply 1×1 convolution to skip connection features Channel compression is performed to obtain semantic representation. ; Sub-step 3-2: Structural branching uses the Sobel operator on the input features Gradient calculations are performed to obtain the edge response map; the edge response map is then processed by a 3×3 convolution followed by a ReLU activation function to obtain an enhanced structural representation. ; Sub-step 3-3: Semantic representation With enhanced structural representation After fusion using 3×3 convolution, the fused features are obtained. ; Sub-steps 3-4: Merging features A gated graph aligned with the spatial location is generated using 1×1 convolution and a sigmoid activation function. ; Sub-steps 3-5: ... With gated graph After element-wise multiplication, combined with learnable balance coefficients Obtain output features .

[0009] In some exemplary embodiments, the will With gated graph After element-wise multiplication, combined with learnable balance coefficients Obtain output features Specifically:

[0010] In the formula, Indicates the learnable balance coefficient; when When the value approaches 1, the module degenerates into an identity mapping; when When the value is close to 0, it becomes more dependent on gated modulation; This represents element-wise multiplication; This indicates the output characteristics after gating modulation.

[0011] In some exemplary embodiments, step 4 specifically includes the following sub-steps: Sub-step 4-1: Process the backbone features output by the Transformer block in the decoder. Perform global average pooling to obtain the channel statistics vector. ; Sub-step 4-2: Convert the channel statistical vector After mapping through two 1×1 convolutional layers and batch normalization layers, the channel-weighted vector is obtained by Softmax normalization. ; Sub-step 4-3: Weight the channel vector Input a Gaussian kernel generator to perform weighted combinations on a set of predefined Gaussian kernels to generate shared low-pass convolution kernels. ; Sub-step 4-4: Using low-pass convolution kernels right Perform a two-dimensional convolution operation to obtain low-frequency features. ; Sub-steps 4-5: ... and Subtraction yields high-frequency features .

[0012] In some exemplary embodiments, step 5 specifically includes the following sub-steps: Sub-step 5-1: Low-frequency features Global max pooling, global standard deviation pooling, and global average pooling are performed, followed by a multilayer perceptron and a sigmoid activation function to generate adaptive weights. These adaptive weights are then applied to... Modulation is performed to obtain the modulated low-frequency characteristics. ; Sub-step 5-2: High-frequency features Global max pooling and global average pooling are performed, followed by 1×7 convolutions, 7×1 convolutions, and a sigmoid activation function to generate adaptive weights. Modulation is performed to obtain the modulated high-frequency characteristics. ; Sub-step 5-3: Perform high-low frequency interaction on the modulated low-frequency and high-frequency features through bidirectional cross-attention to obtain the high-frequency features after low-frequency modulation. and low-frequency characteristics after high-frequency modulation ; Sub-step 5-4: [The sentence is incomplete and requires more context to be translated accurately.] and The fusion process yields the characteristics resulting from the fusion of the modulated high-frequency and low-frequency branches. ; Sub-step 5-5: [The sentence is incomplete and requires more context to be translated accurately.] The signal is transformed to the frequency domain using Fourier transform, and the frequency domain modulation mask is calculated. ; Sub-steps 5-6: ... and After element-wise multiplication, the result is transformed back to the spatial domain by inverse Fourier transform and then combined with the product. Add the residuals together.

[0013] In some exemplary embodiments, step 6 specifically includes the following sub-steps: Sub-step 6-1: Process the input features of each Transformer block after the decoding stage. Perform upsampling operations separately to obtain features. ; Sub-step 6-2: ... By inputting two consecutive Transformer blocks in sequence, the features are obtained. ; Sub-step 6-3: Add features The feature map is converted to an RGB image using a 3×3 convolution, and then compared with the original input degraded image. The residuals are summed to obtain the final restored image.

[0014] According to a second aspect of the present invention, a storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the integrated image restoration method based on frequency modulation and structural prior as described in the first aspect.

[0015] According to a third aspect of the present invention, a computer program product is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, the integrated image restoration method based on frequency modulation and structural prior as described in the first aspect is implemented.

[0016] According to a fourth aspect of the present invention, an electronic device is provided, comprising: Processor; and Memory for storing the executable instructions of the processor; The processor is configured to implement the integrated image restoration method based on frequency modulation and structural prior as described in the first aspect by executing the executable instructions.

[0017] The integrated image restoration method based on frequency modulation and structural prior provided in this invention aims to fully exploit the potential information of images in the frequency domain and achieve adaptive modeling under multiple degradation conditions. This invention designs a frequency decoupling and modulation module, which dynamically separates and fuses high- and low-frequency features through learnable Gaussian kernel generation and residual operations, enhancing the model's ability to jointly represent structure and details. Since existing methods fail to fully exploit the complementary relationship between high- and low-frequency features, this invention designs a bidirectional interactive modulation mechanism, enabling bidirectional guidance and fusion of low-frequency structural information and high-frequency texture information, improving the detail fidelity of image restoration. Furthermore, to suppress the propagation of degradation noise and enhance the transmission of key structural features, this invention proposes a structure-aware gating module, integrating edge priors and gating mechanisms into skip connections. This dynamically filters invalid features and enhances the transmission of valid features, thereby improving the clarity and structural consistency of image restoration.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention. It is obvious that the drawings described below are merely some embodiments of the invention, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0020] Figure 1 This is a flowchart of the method of the present invention.

[0021] Figure 2 This is a structural diagram of the model of the present invention.

[0022] Figure 3 Images for different degradation types.

[0023] Figure 4 The image is restored and reconstructed using the method of the present invention. Detailed Implementation

[0024] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the invention will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0025] Furthermore, the accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0026] To overcome the shortcomings of existing technologies, this invention proposes an integrated image restoration method based on frequency modulation and structural prior. By introducing frequency domain decoupling and adaptive modulation mechanisms, and integrating structurally aware information into the feature modeling process, the method enhances the model's adaptive recovery capability against multiple degradations. Since frequency information is closely related to the structural hierarchy of an image, this invention utilizes frequency modulation mechanisms to extract low-frequency global structural features and high-frequency detailed features. Then, structural priors and frequency features are jointly modeled to achieve a balanced restoration that preserves structure and enhances detail under complex degradation conditions.

[0027] To achieve the above objectives, the present invention adopts the following technical solution: An integrated image restoration method based on frequency modulation and structure prior is proposed, comprising a high-low frequency decoupling network, a bidirectional interactive modulation network, and a structure-aware gating network. Specifically, the high-low frequency decoupling network achieves high-low frequency separation through Gaussian low-pass filtering and residual operation; the bidirectional interactive modulation network introduces bidirectional interaction and frequency domain guidance to achieve complementary modeling of structure and details; and the structure-aware gating network combines structure prior and gating mechanisms in the skip connections between the encoder and decoder to effectively suppress degradation noise and enhance the transfer of effective features.

[0028] refer to Figure 1 As shown, the specific steps may include: Step 1: Perform a 3×3 convolution operation on the input degraded image to extract shallow features. ; Step 2: Shallow features The input is processed in the encoder to extract structural and contextual information at different scales, resulting in multi-scale feature representations. ; Step 3: Introduce structural priors and combine them with gating mechanisms in skip connections to process features from multiple scales. Skip connection features obtained after each Transformer block in the input encoder The modulated output characteristics are obtained. ; Step 4: After encoder processing, the signal enters the decoder stage. A high-low frequency decoupling network is used to process the backbone features output by each Transformer block in the decoder. The process involves decomposing the data into low-frequency features. High frequency characteristics ; Step 5: Utilize a bidirectional interactive modulation network to analyze low-frequency features. High frequency characteristics Enhancement, bidirectional interaction, and frequency domain modulation are performed to obtain fusion features; Step 6: The decoder uses a layer-by-layer upsampling and convolutional reconstruction strategy to map the fused features into a high-quality restored image with the same size as the input.

[0029] The steps in this exemplary embodiment will now be described in more detail with reference to the accompanying drawings and embodiments.

[0030] Step 1: Process low-quality input images Perform a 3×3 convolution operation to extract shallow feature parts. ,in This represents the number of channels. The process is as follows:

[0031] In the formula, This represents the input degraded image; This represents a convolution operation with a kernel size of 3×3; This represents the shallow features obtained after a 3×3 convolution operation.

[0032] Step 2: To extract structural and contextual information at different scales, shallow features are... Multi-stage feature extraction is performed on the input encoder. The encoder consists of four stages, with adjacent stages using convolutional downsampling to halve the resolution and double the number of channels. Therefore, the output feature dimensions change sequentially from stage to stage: , , , The process is as follows:

[0033] In the formula, Indicates the first The Transformer layer contains a multi-head self-attention and feedforward network; Indicates a downsampling operation; Indicates after the first Multi-scale feature representation after extraction from layer Transformer blocks.

[0034] Step 3: Existing methods typically employ direct splicing or additive skip connections between the encoder and decoder. However, encoder features often contain degenerate noise, and direct transmission introduces interference and weakens reconstruction performance. To address this, a structural prior is introduced into the skip connection between the encoder and decoder, combined with a gating mechanism, to effectively suppress invalid or noise components, while allowing useful features to be transmitted to the decoder more stably.

[0035] Sub-step 3-1: Given skip connection features from the encoder This is the sum of features output by each Transformer block in the encoder, which extracts complementary information through two branches. The context feature branch uses a 1×1 convolution to compress the channels, resulting in a semantic representation. The process is as follows:

[0036] In the formula, This represents a convolution operation with a kernel size of 1×1; This represents the semantic representation obtained after passing through the context feature branch.

[0037] Sub-step 3-2: Structural branching uses the Sobel operator on the input features Gradient calculation is performed to explicitly capture structural information such as edges and contours, resulting in an edge response map. This is then aggregated using a 3×3 convolution to gather neighborhood structural responses. The ReLU activation function truncates negative values, preserving significant structural responses and suppressing weak noise, resulting in an enhanced structural representation. , and The spatial dimensions are consistent. The process is as follows:

[0038] In the formula, This represents the edge gradient operator, which calculates the horizontal and vertical gradients for each channel and synthesizes the edge intensity. This indicates that the input features are processed by the Sobel operator. The edge response map obtained after gradient calculation is used to capture the structural and contour information of the image; ReLU represents the linear rectified unit activation function, which is used to enhance the nonlinear expressive power of structural features; This represents the enhanced structural representation obtained through structural branching.

[0039] Sub-step 3-3: Semantic representation obtained from context feature branch Enhanced structural representation obtained from structural branching After 3×3 convolution, local convolution fusion is performed after concatenation to align the two distribution paths and suppress conflicts, resulting in fused features. The process is as follows:

[0040] In the formula, This indicates splicing along the channel dimension; This represents the intermediate feature mapping after fusing contextual semantic features and structural features.

[0041] Sub-steps 3-4: Subsequently, a gated graph aligned with the spatial location is generated using 1×1 convolution and a sigmoid activation function. The value is between (0, 1). The process is represented as follows:

[0042] In the formula, This represents the Sigmoid activation function, used to fuse feature maps. Normalize to the interval (0, 1); This represents the obtained gating graph.

[0043] Sub-steps 3-5: Input features After gating modulation, a learnable coefficient is introduced. To balance the identity mapping and the gated residual output. If If it is a single channel, then in the implementation, the channel will be broadcast to... The same shape. The process is shown below:

[0044] In the formula, Indicates the learnable balance coefficient; when When the value approaches 1, the module degenerates into an identity mapping; when When the value is close to 0, it becomes more dependent on gated modulation; This represents element-wise multiplication; This indicates the output characteristics after gating modulation.

[0045] Step 4: Output the data from sub-steps 3-5 After passing through a 1x1 convolution and a Transformer block, the features are obtained. Through high- and low-frequency decoupling processing, the module decomposes it into low-frequency features and high-frequency features.

[0046] Sub-step 4-1: First process the input features Global average pooling is used to obtain the channel statistics vector. This reflects the global intensity distribution of each channel. The process is as follows:

[0047] In the formula, GAP represents global average pooling, which means averaging over all pixels in each channel; Indicates input features The channel statistics vector obtained by global average pooling.

[0048] Sub-step 4-2: The vector is non-linearly mapped through two 1×1 convolution layers, and a batch normalization layer is used for stable training. After mapping, the weight vector is obtained. The output is then normalized using the Softmax function to ensure numerical stability and interpretability. The process is as follows:

[0049] In the formula, , Represents the learnable convolution weight matrix; This represents the Softmax activation function; This indicates a batch normalization operation; This represents the channel weighted vector.

[0050] Sub-step 4-3: Convert the weight vector Input a Gaussian kernel generator to perform weighted combinations on a set of predefined Gaussian kernels to obtain a shared low-pass convolution kernel. The process is as follows:

[0051] In the formula, The index representing the Gaussian kernel; Indicates the number of Gaussian nuclei; Indicates the first A predefined Gaussian kernel, defined as follows: ; Indicates the first One component is used for control. Weights in the portfolio; Represents the weight vector A low-pass convolution kernel is generated by weighting a set of predefined Gaussian kernels and then using it to smooth the input features to extract their low-frequency components.

[0052] Sub-step 4-4: Apply Gaussian kernel to input features Low-pass convolution is performed to preserve the main structure and smoothing information, resulting in low-frequency features. The process is as follows:

[0053] In the formula, This represents a two-dimensional convolution operation; Represents the initial input features; This indicates the obtained low-frequency characteristics.

[0054] Sub-steps 4-5: Finally, calculate the difference between the original features and the low-frequency components. The high-frequency components are obtained through residual difference analysis. This preserves high-frequency components of the image, such as texture and detail. The process is as follows:

[0055] In the formula, This represents the high-frequency features obtained after difference.

[0056] Step 5: In multi-degraded image restoration, high-frequency and low-frequency features have different effects on global structure and local details. Direct fusion can easily lead to insufficient complementarity between structure and details, and direct amplification of high-frequency noise. Therefore, this invention first enhances high-frequency and low-frequency features separately, then uses bidirectional cross-attention to allow high and low frequencies to complement each other, and finally introduces frequency domain residual modulation to selectively enhance or suppress features at the spectral level, resulting in stable fused features with high information density.

[0057] Sub-step 5-1: Given low-frequency features The low-frequency enhancement unit is used to fuse three types of statistics: global max pooling, global standard deviation pooling, and global average pooling, to improve the channel distribution modeling capability. This is then passed through a multilayer perceptron. Obtain the channel attention map and generate adaptive weight pairs. Modulation is performed. The process is as follows:

[0058] In the formula, This represents a multilayer perceptron; Indicates global max pooling; Indicates global standard deviation pooling; Indicates global average pooling; This represents the Sigmoid activation function, where the weights are in the range (0, 1). This represents element-wise multiplication; This indicates the low-frequency characteristics after modulation by the low-frequency enhancement unit. Sub-step 5-2: In the high-frequency branch, the high-frequency enhancement unit fuses the global max pooling and global average pooling statistics, and then performs directional separable convolution. and The modeling horizontal and vertical spatial orientations are clearly defined, and then fused using 1×1 convolutions to obtain adaptive weight modulation. The process is as follows:

[0059] In the formula, Indicates global max pooling; Indicates global average pooling; This represents the result obtained after global max pooling and global average pooling concat operations on high-frequency features; This indicates a convolution operation with a kernel size of 1×7; This indicates a convolution operation with a kernel size of 7×1. This indicates the high-frequency characteristics after modulation by the high-frequency enhancement unit.

[0060] Sub-step 5-3: High-frequency and low-frequency interaction is achieved through bidirectional cross-attention, allowing high-frequency and low-frequency data to alternate as queries, keys, and values ​​in the cross-attention process. The process is represented as follows:

[0061] In the formula, This represents a cross-attention operation, in the form of: ; This represents the learnable gating coefficient, used to control the intensity of attention modulation, and its range is typically [0,1]. This indicates the high-frequency characteristics after low-frequency modulation; This indicates the low-frequency characteristics after high-frequency modulation.

[0062] Sub-step 5-4: Subsequently, the modulated high-frequency components and low-frequency components are fused, first stitched together on the channel. Then, use a 1×1 convolution to modify the number of channels. Simultaneously, it achieves linear fusion across domains. The process is represented as follows:

[0063] In the formula, This represents the characteristics of the modulated high-frequency branch and low-frequency branch after spatial domain fusion.

[0064] Sub-step 5-5: Merge features The data is transformed to the frequency domain using Fourier operations, and the frequency domain mask is calculated. The process is as follows:

[0065] In the formula, Indicates Fourier operations; Representing the features after fusion The spectral representation obtained by Fourier transform; The amplitude spectrum represents the complex spectrum after the Fourier transform; This indicates logarithmic normalization, which compresses the dynamic range and stabilizes training. Represents the frequency domain response diagram; Indicates the learnable scaling factor; This represents the operation of the hyperbolic tangent function; This represents a learnable frequency domain modulation mask.

[0066] Sub-steps 5-6: Modulate the frequency domain and perform element-wise multiplication. Then, perform an inverse Fourier transform to convert it to the spatial domain, and finally add the residuals to the original fused features to avoid information loss and improve convergence stability. The process is represented as follows:

[0067] In the formula, This indicates the inverse Fourier operation.

[0068] Step 6: In order to generate a high-quality restored image with the same size as the input and clear content, the decoder adopts a layer-by-layer upsampling and convolutional reconstruction strategy to gradually restore the feature space to the original resolution and complete the mapping from the feature domain to the image domain.

[0069] Sub-step 6-1: Analyze the output characteristics after high and low frequency modulation. These will serve as the input features for each Transformer block. Upsampling is performed to the original resolution. The process is as follows:

[0070] In the formula, Representing scale The upsampling operation has a dimension of ; This represents the features obtained after upsampling.

[0071] Sub-step 6-2: The upsampled features are then processed again using a Transformer block. The process is shown below:

[0072] In the formula, and This represents a Transformer block with two consecutive outputs in the final layer. and These represent the features obtained after passing through the Transformer block.

[0073] Sub-step 6-3: Finally, reconstruct the RGB image using a 3×3 convolution, and compare it with the original input degraded image. The residuals are summed to obtain the final high-quality restored image. .

[0074]

[0075] In the formula, Represents the degraded image of the original input; This indicates a high-quality image after restoration.

[0076] like Figure 3 and Figure 4 The image shown is an image before and after reconstruction using the present invention. Figure 4 It can be seen that the present invention can achieve significant quality improvement for different types of degradation: rain streaks are effectively removed, contrast and visibility in hazy scenes are restored, noise interference is significantly reduced, and image details and structural textures are clearly reconstructed. Therefore, the present invention possesses stable restoration capabilities and excellent visual enhancement effects under various degradation conditions.

[0077] It should be noted that, as another aspect, this application also provides a storage medium, which may be included in an electronic device or may exist independently without being assembled into the electronic device. The storage medium carries one or more programs, which, when executed by an electronic device, cause the electronic device to perform the methods described in the following embodiments.

[0078] In one embodiment, this application provides a computer program product including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0079] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0080] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims.

[0081] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is defined only by the appended claims.

Claims

1. An integrated image restoration method based on frequency modulation and structural prior, characterized in that, The method includes: Step 1: Perform a 3×3 convolution operation on the input degraded image to extract shallow features. ; Step 2: Shallow features The input is processed in the encoder to extract structural and contextual information at different scales, resulting in multi-scale feature representations. ; Step 3: Introduce structural priors and combine them with gating mechanisms in skip connections to process features from multiple scales. Skip connection features obtained after each Transformer block in the input encoder The modulated output characteristics are obtained. ; Step 4: After encoder processing, the signal enters the decoder stage. A high-low frequency decoupling network is used to process the backbone features output by each Transformer block in the decoder. The process involves decomposing the data into low-frequency features. High frequency characteristics ; Step 5: Utilize a bidirectional interactive modulation network to analyze low-frequency features. High frequency characteristics Enhancement, bidirectional interaction, and frequency domain modulation are performed to obtain fusion features; Step 6: The decoder uses a layer-by-layer upsampling and convolutional reconstruction strategy to map the fused features into a high-quality restored image with the same size as the input.

2. The method according to claim 1, characterized in that, The encoder in step 2 consists of four stages, with adjacent stages achieving a halving of resolution and a doubling of the number of channels through convolutional downsampling operations.

3. The method according to claim 1, characterized in that, Step 3 specifically includes the following sub-steps: Sub-step 3-1: Apply 1×1 convolution to skip connection features Channel compression is performed to obtain semantic representation. ; Sub-step 3-2: Structural branching uses the Sobel operator on the input features Gradient calculations are performed to obtain the edge response map; the edge response map is then processed by a 3×3 convolution followed by a ReLU activation function to obtain an enhanced structural representation. ; Sub-step 3-3: Semantic representation With enhanced structural representation After fusion using 3×3 convolution, the fused features are obtained. ; Sub-steps 3-4: Merging features A gated graph aligned with the spatial location is generated using 1×1 convolution and a sigmoid activation function. ; Sub-steps 3-5: ... With gated graph After element-wise multiplication, combined with learnable balance coefficients Obtain output features .

4. The method according to claim 3, characterized in that, The With gated graph After element-wise multiplication, combined with learnable balance coefficients Obtain output features Specifically: In the formula, Indicates the learnable balance coefficient; when When the value approaches 1, the module degenerates into an identity mapping; when When the value is close to 0, it becomes more dependent on gated modulation; This represents element-wise multiplication; This indicates the output characteristics after gating modulation.

5. The method according to claim 4, characterized in that, Step 4 specifically includes the following sub-steps: Sub-step 4-1: Process the backbone features output by the Transformer block in the decoder. Perform global average pooling to obtain the channel statistics vector. ; Sub-step 4-2: Convert the channel statistical vector After mapping through two 1×1 convolutional layers and batch normalization layers, the channel-weighted vector is obtained by Softmax normalization. ; Sub-step 4-3: Weight the channel vector Input a Gaussian kernel generator to perform weighted combinations on a set of predefined Gaussian kernels to generate shared low-pass convolution kernels. ; Sub-step 4-4: Using low-pass convolution kernels right Perform a two-dimensional convolution operation to obtain low-frequency features. ; Sub-steps 4-5: ... and Subtraction yields high-frequency features .

6. The method according to claim 5, characterized in that, Step 5 specifically includes the following sub-steps: Sub-step 5-1: Low-frequency features Global max pooling, global standard deviation pooling, and global average pooling are performed, followed by a multilayer perceptron and a sigmoid activation function to generate adaptive weights. These adaptive weights are then applied to... Modulation is performed to obtain the modulated low-frequency characteristics. ; Sub-step 5-2: High-frequency features Global max pooling and global average pooling are performed, followed by 1×7 convolutions, 7×1 convolutions, and a sigmoid activation function to generate adaptive weights. Modulation is performed to obtain the modulated high-frequency characteristics. ; Sub-step 5-3: Perform high-low frequency interaction on the modulated low-frequency and high-frequency features through bidirectional cross-attention to obtain the high-frequency features after low-frequency modulation. and low-frequency characteristics after high-frequency modulation ; Sub-step 5-4: [The sentence is incomplete and requires more context to be translated accurately.] and The fusion process yields the characteristics resulting from the fusion of the modulated high-frequency and low-frequency branches. ; Sub-step 5-5: [The sentence is incomplete and requires more context to be translated accurately.] The signal is transformed to the frequency domain using Fourier transform, and the frequency domain modulation mask is calculated. ; Sub-steps 5-6: ... and After element-wise multiplication, the result is transformed back to the spatial domain by inverse Fourier transform and then combined with the product. Add the residuals together.

7. The method according to claim 1, characterized in that, Step 6 specifically includes the following sub-steps: Sub-step 6-1: Process the input features of each Transformer block after the decoding stage. Perform upsampling operations separately to obtain features. ; Sub-step 6-2: ... By inputting two consecutive Transformer blocks in sequence, the features are obtained. ; Sub-step 6-3: Add features The feature map is converted to an RGB image using a 3×3 convolution, and then compared with the original input degraded image. The residuals are summed to obtain the final restored image.

8. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the integrated image restoration method based on frequency modulation and structural prior as described in any one of claims 1 to 7.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the integrated image restoration method based on frequency modulation and structural prior as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the integrated image restoration method based on frequency modulation and structural priors as described in any one of claims 1 to 7 by executing the executable instructions.

Images