Image restoration method and related device

Abstract

The application discloses an image restoration method and related equipment, and the method comprises the following steps: inputting a to-be-restored degraded image into an image block embedding module to obtain shallow image features; inputting the shallow image features into an encoder to obtain multi-scale coding features; the encoder is composed of a plurality of coding modules, a double-channel representation module in the coding module comprises a structure channel and a detail channel, the structure channel is used for extracting structure information of an image, and the detail channel is used for extracting detail information of the image; inputting the multi-scale coding features into a decoder to obtain target decoding features; inputting the target decoding features into a refinement module to obtain refined features, and mapping the refined features to an image space to obtain a predicted image; and inputting the predicted image and the to-be-restored degraded image into a global residual fusion module to obtain a target restored image. The application can improve the structure restoration capability and the detail restoration capability of the degraded image, improve the restoration precision of the degraded image, and can be widely applied to the technical field of image processing.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image restoration method and related equipment. Background Technology

[0002] Images are easily affected by environmental factors, equipment, and motion during acquisition and transmission, resulting in degradation such as noise, blurring, fog, rain, snow, and low light. The goal of image restoration is to recover a clear image from a degraded image.

[0003] In related technologies, there are unified image restoration methods. These methods employ a shared feature extraction network to map different degraded images to the same feature space, and then use prompts, degradation coding, expert modules, or adaptive modulation modules to generate restored images. However, current unified image restoration methods typically learn image structure, texture details, and degradation information simultaneously within the same shared feature space, which mainly suffers from the following problems: global structural information and local detail information are easily mixed; a single feature processing method cannot simultaneously handle large-scale structural restoration and fine texture restoration; and when faced with multiple or compound degradations, the restored image is prone to incomplete structure, blurred texture, or residual degradation.

[0004] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0005] The embodiments of this application aim to at least partially solve one of the technical problems in the related art. Therefore, the main objective of the embodiments of this application is to propose an image restoration method and related equipment that can improve the structural and detail restoration capabilities of degraded images, effectively enhancing the restoration accuracy of degraded images.

[0006] To achieve the above objectives, one aspect of this application proposes an image restoration method, which includes the following steps: The degraded image to be restored is input into the image block embedding module for feature embedding processing to obtain shallow image features; The shallow image features are input into an encoder for feature encoding processing to obtain multi-scale encoded features; wherein, the encoder consists of several encoding modules, and each encoding module consists of a dual-path representation module, which includes a structural path and a detail path. The structural path is used to extract structural information of the image, and the detail path is used to extract detailed information of the image. The multi-scale encoded features are input into the decoder for feature decoding processing to obtain the target decoded features; The target decoding features are input into the refinement module for feature refinement processing to obtain refined features, and the refined features are mapped to the image space to obtain the predicted image; The predicted image and the degraded image to be restored are input into the global residual fusion module for global residual fusion processing to obtain the target restored image.

[0007] In some embodiments, the step of inputting the shallow image features into an encoder for feature encoding processing to obtain multi-scale encoded features includes: The shallow image features are input into the encoder, and the first encoding module in the encoder is determined as the current encoding module, and the shallow image features are determined as the current image input features; The dual-path representation module in the current encoding module performs complementary representation of the image structure information and image detail information in the current image input features to obtain the current intermediate representation result. If the current encoding module is not the last encoding module in the encoder, then the next encoding module in the encoder is determined as the current encoding module, the current intermediate representation result is determined as the current image input feature, and the process of performing complementary representation on the image structure information and image detail information in the current image input feature through the dual-path representation module in the current encoding module to obtain the current intermediate representation result is repeated until the current encoding module is the last encoding module in the encoder, and the current intermediate representation result output by the last encoding module is determined as the deep encoding feature of the encoder. The multi-scale coding features are constructed based on the current intermediate representation results output by each of the coding modules.

[0008] In some embodiments, the step of performing complementary representation of image structure information and image detail information in the current image input features through the dual-path representation module in the current encoding module to obtain the current intermediate representation result includes: The current image input features are normalized through the first normalization layer in the dual-path representation module to obtain the first normalized features; The structural path features are obtained by performing feature transformation operations on the first normalized features through the structural path in the dual-path characterization module. By using the detail path in the dual-path characterization module, feature transformation operation is performed on the first normalized feature to obtain the detail path feature; The dual-path fusion unit in the dual-path representation module performs element-wise residual addition on the structural path features, the detail path features, and the current image input features to obtain the dual-path fusion features. The dual-path fusion features are normalized through the second normalization layer in the dual-path representation module to obtain the second normalized features. The frequency-aware refinement module in the dual-path characterization module refines the second normalized feature to obtain the frequency-aware refined feature. The current intermediate representation result is obtained by adding the element-wise residuals of the frequency-aware refinement feature and the dual-path fusion feature.

[0009] In some embodiments, the step of performing feature transformation on the first normalized feature through the structural pathway in the dual-pathway representation module to obtain structural pathway features includes: The first normalized feature is subjected to feature projection processing through the input convolutional unit in the structural path to obtain the structural projection feature. Local spatial information is extracted from the structural projection features by using the deep convolutional units in the structural pathway to obtain structural spatial aggregation features; The structural spatial aggregation features are subjected to complementary feature partitioning to obtain a first feature group and a second feature group; The first feature group and the second feature group are exchanged through the bidirectional feature interaction unit in the structural path to obtain the first interactive feature and the second interactive feature. By performing element-by-element interaction on the first interactive feature and the second interactive feature, a bidirectional interactive fusion feature is obtained; The output convolutional units in the structural pathway are used to perform channel mapping and feature aggregation on the bidirectional interactive fusion features to obtain the structural pathway features.

[0010] In some embodiments, the step of performing feature transformation on the first normalized feature through the detail path in the dual-path representation module to obtain detail path features includes: The first normalized feature is processed by channel mapping through the input projection unit in the detail path to obtain the detail projection feature. The Gaussian difference frequency refinement unit in the detail path performs Gaussian difference frequency decomposition on the detail projection features to obtain frequency features. The frequency detail features are obtained by generating adaptive gating information based on the detail projection features through the frequency band gating unit in the detail path, and then modulating the frequency features element by element based on the adaptive gating information. The global context extraction unit in the detail path performs global spatial information aggregation on the detail projection features to obtain global context features. The global context features and the frequency detail features are spliced ​​together by the feature splicing and fusion unit in the detail path to obtain the fused detail features; The fused detail features are obtained by channel mapping and feature aggregation through the output projection unit in the detail path.

[0011] In some embodiments, the step of refining the second normalized feature through the frequency-aware refinement module in the dual-path characterization module to obtain the frequency-aware refined feature includes: The second normalized feature is processed by the input projection unit in the frequency sensing refinement module to obtain the interactive projection feature. The interactive projection features are subjected to feature splitting processing to obtain a first feature branch and a second feature branch; The first feature branch is locally extracted by using the deep convolutional unit in the frequency-aware thinning module to obtain the locally extracted features. The local extracted features are nonlinearly activated by the nonlinear activation unit in the frequency sensing refinement module to generate adaptive modulation information. The adaptive modulation information and the feature response of the second feature branch are multiplied element-wise by the element-wise multiplication unit in the frequency sensing refinement module to obtain the gated enhancement feature. The output projection unit in the frequency sensing refinement module performs channel mapping and feature aggregation on the gated enhancement features to obtain mapped and aggregated features. The frequency-aware refinement feature is obtained by refining the mapped aggregated feature in the local frequency domain through the local image block frequency domain processing unit in the frequency-aware refinement module.

[0012] To achieve the above objectives, another aspect of this application provides an image restoration apparatus, which includes the following modules: The feature embedding processing module is used to input the degraded image to be restored into the image block embedding module for feature embedding processing to obtain shallow image features; The feature encoding processing module is used to input the shallow image features into the encoder for feature encoding processing to obtain multi-scale encoded features; wherein, the encoder is composed of several encoding modules, and each encoding module is composed of a dual-path representation module, which includes a structural path and a detail path. The structural path is used to extract the structural information of the image, and the detail path is used to extract the detail information of the image. The feature decoding processing module is used to input the multi-scale encoded features into the decoder for feature decoding processing to obtain the target decoded features; The feature refinement module is used to input the target decoded features into the refinement module for feature refinement processing to obtain refined features, and to map the refined features to the image space to obtain the predicted image; The global residual fusion module is used to input the predicted image and the degraded image to be restored into the global residual fusion module for global residual fusion processing to obtain the target restored image.

[0013] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0014] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0015] To achieve the above objectives, another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0016] The embodiments of this application include at least the following beneficial effects: This application provides an image restoration method and related equipment. This scheme involves inputting the degraded image to be restored into an image block embedding module for feature embedding processing to obtain shallow image features; inputting the shallow image features into an encoder for feature encoding processing to obtain multi-scale encoded features; wherein, the encoder consists of several encoding modules, and the encoding module consists of a dual-path representation module, which includes a structural path and a detail path. The structural path is used to extract the structural information of the image, and the detail path is used to extract the detail information of the image; the multi-scale encoded features are input into a decoder for feature decoding processing to obtain target decoded features; the target decoded features are input into a thinning module for feature thinning processing to obtain thinned features, and the thinned features are mapped to the image space to obtain a predicted image; the predicted image and the degraded image to be restored are input into a global residual fusion module for global residual fusion processing to obtain the target restored image. This application embodiment extracts complete shallow image features through image patch embedding, providing reliable basic feature support for subsequent feature encoding and image restoration. An encoder containing a dual-path representation module performs multi-scale encoding of shallow features, independently extracting global structural information and local detail information using structural and detail paths respectively, reducing mutual interference between different types of image information in the same processing path. A decoder completes the decoding and reconstruction of multi-scale features, ensuring the complete restoration of multi-scale feature information. A refinement module further optimizes feature representation capabilities, improving image prediction quality. Global residual fusion fully combines the original degraded image information with the predicted image information, compensating for information loss during feature restoration. Therefore, this application embodiment can simultaneously restore the overall structure of the object and the texture details, thereby improving the structural and detail restoration capabilities of degraded images and effectively enhancing the restoration accuracy of degraded images. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the steps of an image restoration method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a dual-path characterization module provided in an embodiment of this application; Figure 3 This is a schematic diagram of a structural pathway for learning image structural information provided in an embodiment of this application; Figure 4 This is a schematic diagram of a detail path for learning frequency detail information provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a frequency sensing refinement module provided in an embodiment of this application; Figure 6 This is a schematic diagram of the overall structure of a cortical dual-pathway image restoration network provided in an embodiment of this application; Figure 7This is a schematic diagram of the structure of an image restoration device provided in an embodiment of this application; Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application.

[0019] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0020] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0022] In practical applications, images are easily affected by environmental factors, equipment, and motion during acquisition and transmission, resulting in degradation such as noise, blurring, fog, rain, snow, and low light. The goal of image restoration is to recover a clear image from a degraded image.

[0023] Early image restoration methods typically built specialized models for specific types of degradation. For example, denoising models primarily dealt with image noise, deblurring models primarily dealt with motion blur, and dehazing models primarily addressed image quality degradation caused by atmospheric scattering. With the development of deep learning technology, convolutional neural networks and Transformer networks have been widely used in image restoration. These methods can automatically learn the mapping relationship between degraded images and clear images through training data.

[0024] However, real-world images may contain multiple types of degradation, or even two or more types of degradation simultaneously. Therefore, using a separate model to handle each type of degradation would increase the cost of model deployment, storage, and maintenance. To address this, related technologies have begun to adopt a unified image restoration model, using a single model to handle multiple degradations. Currently, related unified image restoration methods typically employ a shared feature extraction network to map different degraded images to the same feature space, and then use prompts, degradation encoding, expert modules, or adaptive modulation modules to generate restored images. Representative related schemes include: (1) AirNet: Identifies unknown degradations through degradation-aware representations and completes unified image restoration. (2) PromptIR: Uses learnable prompts to enable the network to adjust image restoration features according to different degradations. (3) AdaIR: Uses frequency mining and adaptive modulation to handle multiple image degradations. (4) OneRestore: Establishes a unified restoration framework for composite degraded images with multiple degradations coexisting.

[0025] However, current unified image restoration methods typically learn image structure, texture details, and degradation information simultaneously within the same shared feature space, which mainly suffers from the following problems: global structural information and local detail information are easily mixed; a single feature processing method cannot simultaneously handle large-scale structural restoration and fine texture restoration; when faced with multiple or compound degradations, the restored image is prone to incomplete structure, blurred texture, or residual degradation; directly performing feature processing in the spatial domain may not fully utilize the differences between different degradations in the frequency domain; and building separate models for different degradations increases the number of models and deployment costs.

[0026] In view of this, this application provides an image restoration method and related apparatus. This scheme extracts complete shallow image features through image patch embedding, providing reliable basic feature support for subsequent feature encoding and image restoration. An encoder containing a dual-path representation module performs multi-scale encoding of shallow features, independently extracting global structural information and local detail information using structural and detail paths respectively, reducing mutual interference between different types of image information in the same processing path. A decoder completes the decoding and reconstruction of multi-scale features, ensuring the complete restoration of multi-scale feature information. A refinement module further optimizes feature representation capabilities, improving image prediction quality. Global residual fusion fully combines the original degraded image information with the predicted image information, compensating for information loss during feature restoration. Therefore, this application can simultaneously restore the overall structure of the object and the texture details, thereby improving the structural and detail restoration capabilities of degraded images and effectively enhancing the restoration accuracy of degraded images.

[0027] This application provides an image restoration method, which relates to the field of image processing technology. The image restoration method provided in this application can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing an image restoration method, but is not limited to the above forms.

[0028] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0029] Please see Figure 1 , Figure 1 This is an optional flowchart of an image restoration method provided in an embodiment of this application. Figure 1 The method may include, but is not limited to, steps S101 to S105.

[0030] Step S101: Input the degraded image to be restored into the image block embedding module for feature embedding processing to obtain shallow image features; Degraded images are those that have been affected by environmental factors, equipment, motion, etc., during acquisition and transmission, resulting in one or more degradations such as noise, blurring, fog, rain, snow, and low illumination. In this embodiment, the degraded image to be restored is the degraded image that requires restoration processing.

[0031] Image restoration refers to the process of recovering a degraded image affected by factors such as noise, blur, fog, rain, snow, or low light into a clear image. Unified image restoration, on the other hand, uses the same image restoration model to handle multiple different types of image degradation.

[0032] The feature embedding process includes local feature extraction and feature mapping. Local feature extraction is used to extract low-level local visual information such as edges, contours, and local textures within an image patch, on a patch-by-patch basis. Feature mapping is used to map the extracted local features to a feature space of a preset dimension to obtain features of a fixed channel dimension.

[0033] Optionally, the shallow image features are the features obtained after local feature extraction and feature mapping of the degraded image to be restored. These shallow image features retain low-level visual information such as image edges, contours, and local textures, and are used as input features for the encoder.

[0034] Step S102: Input the shallow image features into the encoder for feature encoding processing to obtain multi-scale encoded features; wherein, the encoder consists of several encoding modules, and each encoding module consists of a dual-path representation module, which includes a structural path and a detail path. The structural path is used to extract the structural information of the image, and the detail path is used to extract the detail information of the image. In some embodiments, step S102 may include: inputting shallow image features into the encoder, determining the first encoding module in the encoder as the current encoding module, and determining the shallow image features as the current image input features; performing complementary representation on the image structure information and image detail information in the current image input features through the dual-path representation module in the current encoding module to obtain the current intermediate representation result; if the current encoding module is not the last encoding module in the encoder, determining the next encoding module in the encoder as the current encoding module, determining the current intermediate representation result as the current image input feature, and returning to execute the step of performing complementary representation on the image structure information and image detail information in the current image input features through the dual-path representation module in the current encoding module to obtain the current intermediate representation result, until the current encoding module is the last encoding module in the encoder, and determining the current intermediate representation result output by the last encoding module as the deep encoding feature of the encoder; constructing multi-scale encoding features based on the current intermediate representation results output by each encoding module.

[0035] For the current image input features, these are the features input to the current encoding module. In the first round of encoding, the current image input features are shallow image features. Subsequent encoding modules use the current intermediate representation results output by the previous level encoding module as the current image input features. Specifically, the current intermediate representation results are the features output by the dual-path representation module within a single encoding module after completing complementary representations of the current image input features. These features serve as the input features for the next level encoding module and are also used to construct multi-scale encoded features.

[0036] The multi-scale coding features are composed of the current intermediate representation results output by each coding module of the encoder. Different coding modules correspond to different receptive fields and different resolution scales. The current intermediate representation results output by each coding module are collected to obtain the multi-scale coding features, which include shallow scale features, medium scale features, and deep coding features.

[0037] Optionally, the deep coding feature is the current intermediate representation result output by the last coding module in the encoder, which has the largest receptive field, and serves as the initial input feature of the decoder.

[0038] In some specific embodiments, the step of performing complementary representation of image structure information and image detail information in the current image input features through the dual-path representation module in the current encoding module to obtain the current intermediate representation result may include: normalizing the current image input features through the first normalization layer in the dual-path representation module to obtain the first normalized feature; performing feature transformation operation on the first normalized feature through the structural path in the dual-path representation module to obtain the structural path feature; performing feature transformation operation on the first normalized feature through the detail path in the dual-path representation module to obtain the detail path feature; performing element-wise residual summation of the structural path feature, detail path feature, and current image input feature through the dual-path fusion unit in the dual-path representation module to obtain the dual-path fusion feature; normalizing the dual-path fusion feature through the second normalization layer in the dual-path representation module to obtain the second normalized feature; performing feature refinement of the second normalized feature through the frequency-aware refinement module in the dual-path representation module to obtain the frequency-aware refinement feature; and performing element-wise residual summation of the frequency-aware refinement feature and the dual-path fusion feature to obtain the current intermediate representation result.

[0039] The encoder consists of multiple encoding modules (or encoding stages), and each encoding module (or encoding stage) consists of one or more dual-path representation modules.

[0040] For structural path features, these are the features output after the structural path performs a feature transformation operation on the first normalized feature, including global structural and contextual information such as image object contours, region layout, and overall shape. For detail path features, these are the features output after the detail path performs a feature transformation operation on the first normalized feature, including local details and frequency information such as image edges, textures, and fine-grained high frequencies. For dual-path fusion features, these are the features obtained by adding the structural path features, detail path features, and the current image input features element-wise residuals, fusing the original input information, global structural information, and local detail information to achieve complementary representation of structural and detail information. For frequency-aware refinement features, these are the features obtained after the frequency-aware refinement module performs feature refinement processing on the second normalized feature, used to enhance the effective frequency information of the image and suppress interference caused by degradation.

[0041] Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of a dual-path characterization module provided in an embodiment of this application, as shown below. Figure 2As shown, the dual-path representation module includes a first normalization layer, a structural path, a detail path, a dual-path fusion unit, a second normalization layer, and a frequency-aware refinement module. The structural path, also known as the M-path, is a feature processing branch used to extract global context and overall structural information of the image. Specifically, it extracts structural information such as object contours, region layout, brightness variations, and overall shape. The detail path, also known as the P-path, is a feature processing branch used to extract frequency information, texture, and local details of the image. Specifically, it extracts detail information such as edges, textures, and other fine-grained high-frequency information. High-frequency information typically corresponds to edges, textures, and subtle variations in the image.

[0042] In specific implementations, such as Figure 2 As shown, the specific implementation process of the dual-path representation module for complementary representation of image structural information and image detail information is as follows: The current image input features are first normalized through the first normalization layer to obtain the first normalized features. These first normalized features are then input into the structural path and the detail path, respectively. The structural path outputs structural path features, and the detail path outputs detail path features. Then, the structural path performs feature aggregation or projection operations on the input first normalized features to extract global image structural information and outputs structural path features. The detail path performs feature aggregation or projection operations on the input first normalized features to extract local fine texture information and outputs detail path features. The two paths, through their respective feature aggregation or projection operations, ensure that the structural features and detail features have the same spatial size and channel dimension. Subsequently, the structural path features, detail path features, and the original input features of the dual-path representation module (i.e., the current image input features) are added element-wise according to their corresponding spatial positions and corresponding channels to obtain the dual-path fusion features. The dual-path fusion process is represented as follows: ; in, This represents the input features of the dual-path characterization module. The structural path characteristics representing the output of the structural path. The detail path features represent the output of the detail path. The fused complementary features are represented as dual-path fusion features. Through element-wise residual fusion, the overall structure and context information obtained from the structural path and the edge, texture, and frequency detail information obtained from the detail path are jointly represented while preserving the original input information. Next, the dual-path fusion features are normalized through a second normalization layer to obtain second normalized features. These second normalized features are then input into a frequency-aware refinement module for further feature refinement, resulting in frequency-aware refined features. Finally, the frequency-aware refined features and the dual-path fusion features are again added element-wise to obtain the final output features of the dual-path representation module.

[0043] In some specific embodiments, the step of performing feature transformation operations on the first normalized feature through the structural path in the dual-path representation module to obtain structural path features may include: performing feature projection processing on the first normalized feature through the input convolutional unit in the structural path to obtain structural projection features; extracting local spatial information from the structural projection features through the depth convolutional unit in the structural path to obtain structural spatial aggregation features; performing complementary feature partitioning processing on the structural spatial aggregation features to obtain a first feature group and a second feature group; performing information exchange processing on the first feature group and the second feature group through the bidirectional feature interaction unit in the structural path to obtain a first interactive feature and a second interactive feature; performing element-wise interaction on the first interactive feature and the second interactive feature to obtain bidirectional interactive fusion features; and performing channel mapping and feature aggregation on the bidirectional interactive fusion features through the output convolutional unit in the structural path to obtain structural path features.

[0044] Among them, the first feature group and the second feature group are two sets of features obtained after performing complementary feature partitioning on the aggregated features of the structural space. The two sets of features are complementary to each other and are used to complete the bidirectional information transmission within the bidirectional feature interaction unit.

[0045] Specifically, the bidirectional feature interaction unit includes an interaction branch from the first feature group to the second feature group and an interaction branch from the second feature group to the first feature group. In the first-direction feature interaction process, the first feature group generates an interaction response related to structural information, and modulates and updates the second feature group, allowing the structural information in the first feature group to be transmitted to the second feature group, outputting the first interaction feature. In the second-direction feature interaction process, feature compression and correlation calculation are performed on the two feature groups to establish a relationship between them. This relationship is then used to modulate and update the first feature group, allowing the contextual information in the second feature group to be transmitted back to the first feature group, outputting the second interaction feature.

[0046] For the first interactive feature, it is the feature output by the bidirectional feature interaction unit in the first interaction direction after transmitting the structural information of the first feature group and modulating and updating the second feature group. For the second interactive feature, it is the feature output by the bidirectional feature interaction unit in the second interaction direction after transmitting the context information of the second feature group in reverse and modulating and updating the first feature group.

[0047] For a detailed implementation, please refer to Figure 3 , Figure 3 This is a schematic diagram of a structural pathway for learning image structural information provided in an embodiment of this application, such as... Figure 3 As shown, the structural path includes an input convolutional unit, a depthwise convolutional unit, a bidirectional feature interaction unit, and an output convolutional unit. The specific operation flow of the structural path is as follows: The structural path receives... Figure 2 The first normalized feature output by the first normalization unit in the dual-path representation module is used as input. Specifically, the input first normalized feature is first projected through the input convolution unit of the structural path to obtain structural projection features. Then, through the depth convolution unit of the structural path, local spatial information is further extracted from the structural projection features to obtain intermediate features (i.e., the aforementioned structural spatial aggregation features) used for subsequent structural relationship modeling. Next, the intermediate features are divided into two complementary feature groups, denoted as the first feature group and the second feature group, respectively. Information exchange occurs between the two feature groups through the bidirectional feature interaction unit of the structural path, resulting in the first interactive feature and the second interactive feature. After completing the bidirectional interaction, the interactive features obtained from the two directions are interacted element-wise according to their corresponding spatial and channel positions to jointly retain the structural responses in the two complementary feature groups, resulting in bidirectional interactive fusion features. Subsequently, the output convolution unit of the structural path performs channel mapping and feature aggregation on the bidirectional interactive fusion features to obtain the structural path features output by the structural path. Through the above bidirectional information exchange, cross-regional structural associations can be established between different feature groups, thereby enhancing the representation ability of the overall image contour, spatial structure, and contextual dependencies.

[0048] In some specific embodiments, the step of performing feature transformation on the first normalized feature through the detail path in the dual-path representation module to obtain the detail path feature may include: performing channel mapping processing on the first normalized feature through the input projection unit in the detail path to obtain the detail projection feature; performing Gaussian difference frequency decomposition processing on the detail projection feature through the Gaussian difference frequency refinement unit in the detail path to obtain the frequency feature; generating adaptive gating information based on the detail projection feature through the frequency band gating unit in the detail path, and performing element-wise modulation on the frequency feature based on the adaptive gating information to obtain the frequency detail feature; performing global spatial information aggregation on the detail projection feature through the global context extraction unit in the detail path to obtain the global context feature; splicing the global context feature and the frequency detail feature through the feature splicing and fusion unit in the detail path to obtain the fused detail feature; and performing channel mapping and feature aggregation on the fused detail feature through the output projection unit in the detail path to obtain the detail path feature.

[0049] Among them, the frequency features are the features obtained by the Gaussian difference frequency refinement unit in the detail path after performing Gaussian difference frequency decomposition on the detail projection features. They contain frequency information corresponding to the differences in Gaussian response at multiple scales and are used to characterize image edges, textures, and details at different spatial scales.

[0050] For adaptive gating information, it is the weight information generated by the band gating unit in the detail path based on the detail projection features. It is used to perform element-wise modulation with the frequency features, selectively enhancing the effective frequency response and suppressing the ineffective response.

[0051] For a detailed implementation, please refer to Figure 4 , Figure 4 This is a schematic diagram of a detail path for learning frequency detail information provided in an embodiment of this application, such as... Figure 4 As shown, the detail path includes an input projection unit, a Gaussian difference frequency refinement unit, a frequency band gating unit, a global context extraction unit, a feature stitching and fusion unit, and an output projection unit. The specific operation flow of the detail path is as follows: detail path receives... Figure 2 The first normalized feature output by the first normalization unit in the first layer is used as input; specifically, the first normalized feature is first channel-mapped by the input projection unit of the detail path to obtain the detail projection feature. The detail projection features are then input into the frequency refinement branch, the gating branch, and the global context branch, respectively. In the frequency refinement branch, the detail projection features are subjected to Gaussian difference frequency decomposition, i.e., Difference of Gaussian (DoG) processing, through the Difference of Gaussian frequency refinement unit. The Gaussian difference frequency decomposition function is used. This Gaussian difference frequency refinement unit obtains multiple frequency responses based on the differences in Gaussian responses at different scales using this function. These multiple frequency responses are then combined to form frequency features containing components at multiple spatial scales. These frequency features carry details about edges, textures, and different spatial scales in the image. In the gating branch, the bandgap gating unit uses the bandgap gating function based on the detail projection features. An adaptive gating information corresponding to the multi-scale frequency response output of the Gaussian difference frequency decomposition function is generated. This adaptive gating information is then used to modulate the frequency features generated by the frequency refinement branch element-wise, thereby selectively enhancing the effective frequency response and suppressing the ineffective response to obtain the frequency detail features. , For element-wise multiplication, in parallel with the frequency refinement branch, the detail projection features are input into the global context branch. The global context extraction unit then aggregates the spatial information of the detail projection features to obtain global context features that contain a large range of scene content and contextual relationships. , For the global context extraction module, global context features This is used to supplement the overall content information that may be missing when using only local frequency information; subsequently, the frequency detail features obtained through gating modulation are... With global context features The features are concatenated along the feature dimension to obtain fused detail features. Finally, the concatenated fused detail features are channel-mapped and feature-aggregated using the output projection unit to obtain the detail path features of the detail path output. ,in, To output the projection function, This is a feature splicing operation along the channel dimension.

[0052] In some specific embodiments, the step of refining the second normalized features using the frequency-aware refinement module in the dual-path representation module to obtain frequency-aware refined features may include: performing feature mapping on the second normalized features through the input projection unit in the frequency-aware refinement module to obtain interactive projection features; performing feature splitting on the interactive projection features to obtain a first feature branch and a second feature branch; performing local feature extraction on the first feature branch through the depth convolution unit in the frequency-aware refinement module to obtain locally extracted features; performing nonlinear activation on the locally extracted features through the nonlinear activation unit in the frequency-aware refinement module to generate adaptive modulation information; multiplying the adaptive modulation information and the feature responses of the second feature branch element-wise through the element-wise multiplication unit in the frequency-aware refinement module to obtain gated enhancement features; performing channel mapping and feature aggregation on the gated enhancement features through the output projection unit in the frequency-aware refinement module to obtain mapped aggregated features; and refining the mapped aggregated features in the local frequency domain through the local image block frequency domain processing unit in the frequency-aware refinement module to obtain frequency-aware refined features.

[0053] Among them, the adaptive modulation information is the information generated by the nonlinear activation unit in the frequency sensing refinement module after performing nonlinear activation processing on the locally extracted features. It is used to adjust the response intensity of different spatial locations and channels to achieve dynamic feature weighting.

[0054] The gating enhancement feature is obtained by multiplying the adaptive modulation information with the feature response of the second feature branch element by element by element by the element-wise multiplication unit in the frequency sensing refinement module. It can enhance the structural and detailed features of the high effective response and weaken the feature response that is not highly relevant to the recovery task.

[0055] For a detailed implementation, please refer to Figure 5 , Figure 5 This is a schematic diagram of the structure of a frequency sensing refinement module provided in an embodiment of this application, as shown below. Figure 5 As shown, the frequency-aware thinning module includes an input projection unit, a depthwise convolution unit, a nonlinear activation unit, an element-wise multiplication unit, an output projection unit, and a local image patch frequency domain processing unit. The specific operation flow of the frequency-aware thinning module is as follows: The frequency-aware thinning module receives... Figure 2The second normalized feature output from the second-layer normalization unit in the dual-path representation module is used as input. The input second normalized feature is first processed by the input projection unit for feature mapping to obtain interactive projection features. Then, the interactive projection features undergo feature splitting to form two corresponding feature branches. The first feature branch extracts local features through depthwise convolution and then generates adaptive modulation information to adjust the response intensity of different spatial locations and channels through the nonlinear activation unit GELU. The second feature branch retains the original projection feature response with the same dimension as the first feature branch. Subsequently, the adaptive modulation information generated by the first feature branch is multiplied element-wise with the feature response of the second feature branch according to the corresponding spatial location and channel. Thus, the adaptive modulation information dynamically weights the different feature responses in the second feature branch, enhancing structural and detailed features with high effective responses while weakening feature responses with low relevance to the current recovery task, resulting in gated enhancement features. Next, the gated enhancement features are further input to the output projection unit for channel mapping and feature aggregation to obtain mapped and aggregated features. These features are then processed by the local image patch frequency domain processing unit using the local image patch fast Fourier transform (Patch). The FFT (Focus-Free Fourier Transform) refines the mapped aggregated features in the local frequency domain, ultimately yielding the output features of the frequency-aware refinement module, i.e., the frequency-aware refinement features.

[0056] The operation process of the frequency sensing refinement module can be represented by the following calculation formula: ; ; ; ; in, For module input features The projection features obtained after performing the input projection operation. The input features are for the frequency-aware refinement module. and These are two complementary feature branches formed after the input is projected. For the input projection function, For depthwise convolution functions, The GELU nonlinear activation function is used. These are the features after gating adaptive enhancement. The output features of the frequency-sensing feature module. To output the projection function, This is the Fast Fourier Transform function for a local image patch.

[0057] Step S103: Input the multi-scale encoded features into the decoder for feature decoding processing to obtain the target decoded features; The decoder includes multiple cascaded dual-path representation modules and multiple feature fusion modules, with the feature fusion modules positioned between adjacent dual-path representation modules. The initial features received by the decoder are first input to the first dual-path representation module. The features output by this module are upsampled and then fed into the corresponding feature fusion module. The features output by the feature fusion module serve as the input to the next-level dual-path representation module. This process is performed level by level until the last-level dual-path representation module directly outputs the decoded features. It should be noted that the number of dual-path representation modules and feature fusion modules in the decoder can be set according to the actual application; this embodiment does not impose any limitations on this.

[0058] In the specific implementation, after encoding is completed, the multi-scale encoded features are sent to the decoder. The deep encoded features in the multi-scale encoded features are used as the initial input of the decoder. The operations performed in the decoder include: first, the deep encoded features are input into the first dual-path representation module in the decoder for feature reconstruction; then, the decoder upsamples the current features output by the first dual-path representation module so that the resolution of the upsampled features matches the encoded features of the encoder at the corresponding scale in the multi-scale encoded features; and then, the upsampled features and the encoded features of the encoder at the corresponding scale in the multi-scale encoded features are fused. Next, the features after the first round of fusion are input into the next level dual-path representation module for reconstruction, and the upsampling, feature fusion, and reconstruction operations are repeated on the output features of the next level dual-path representation module until the reconstructed features of the last dual-path representation module are output. In this way, the spatial resolution of the features is restored step by step to obtain the decoded features.

[0059] It should be noted that the internal processing of the dual-path representation module in the decoder is the same as that of the dual-path representation module described in step S102 above, and will not be repeated here in the embodiments of this application.

[0060] In addition, the upsampling operation is used to improve the spatial resolution of the feature map; the feature fusion operation is used to fuse the upsampled features with the corresponding scale encoded features output by the encoder, which can be achieved by feature concatenation or pixel-by-pixel addition; the specific implementation principles of the upsampling and feature fusion operations can be referred to the relevant technologies in this technical field, and this application embodiment does not limit them, and this application embodiment will not elaborate on them here.

[0061] Step S104: Input the target decoding features into the refinement module for feature refinement processing to obtain refined features, and map the refined features to the image space to obtain the predicted image; In a specific implementation, the decoded features are further input into a refinement module. The refinement module is composed of one or more dual-path representation modules, which further enhance the structural and detail representation while maintaining the feature space resolution. Its internal processing is the same as that of the dual-path representation module described in step S102 above, and will not be repeated here. After the refinement module completes feature refinement and obtains refined features through the dual-path representation module, the refined features are mapped to the image pixel space inside the refinement module. The high-dimensional features are converted to image pixel representation through convolution operation, the channel and pixel value features of the target image are matched, and finally the predicted image is generated. The specific implementation principle of this feature mapping operation can be referred to the relevant technologies in this technical field. This application does not limit this, and will not be repeated here.

[0062] Step S105: Input the predicted image and the degraded image to be restored into the global residual fusion module for global residual fusion processing to obtain the target restored image.

[0063] In the specific implementation, after mapping the refined features to the image space to obtain the predicted image, the global residual fusion module performs global residual fusion between the predicted image mapped to the image space and the input degraded image to obtain the restored clear image (i.e. the target restored image), and the restored clear image is output through the output image module.

[0064] Steps S101 to S105 of this embodiment extract complete shallow image features through image patch embedding, providing reliable basic feature support for subsequent feature encoding and image restoration. The shallow features are encoded at multiple scales using an encoder containing a dual-path representation module. Global structural information and local detail information are extracted independently using structural and detail paths, reducing interference between different types of image information in the same processing path. The decoder completes the decoding and reconstruction of multi-scale features, ensuring complete restoration of multi-scale feature information. The refinement module further optimizes feature representation capabilities, improving image prediction quality. Global residual fusion fully combines the original degraded image information with the predicted image information, compensating for information loss during feature restoration. Therefore, this embodiment can simultaneously restore the overall structure of the object and the texture details, thereby improving the structural and detail restoration capabilities of degraded images and effectively enhancing the restoration accuracy of degraded images.

[0065] To explain in detail the principles of the technical solution of this application, the overall process of this application will be described below with reference to some specific embodiments. It is easy to understand that the following is an explanation of the technical principles of this application and should not be regarded as a limitation of this application.

[0066] Related technical solutions typically do not explicitly define two dedicated functional pathways responsible for global structure and local details, respectively. This application, however, establishes a structural pathway for extracting global structural information and a detail pathway for extracting local detail information. A frequency-aware module is used to fuse and optimize the two types of features output from these two pathways, thereby improving the structural and detail recovery capabilities of degraded images. Furthermore, this application can simultaneously train a single image restoration model to learn degradation features from different scenarios, enabling a single model to handle multiple types of image degradation without requiring the deployment of multiple separate models. This reduces the cost of deploying separate models for different degradation types by using a single image restoration model to handle one or more types of image degradation.

[0067] Please see Figure 6 , Figure 6 This is a schematic diagram of the overall structure of a cortical dual-pathway image restoration network provided in an embodiment of this application, as shown below. Figure 6 As shown, this application provides a cortical dual-pathway image restoration network, which includes an input image module, an image patch embedding module, an encoder, a decoder, a thinning module, a global residual fusion module, and an output image module. Figure 6As shown, the overall image restoration process based on the cortical dual-pathway image restoration network is as follows: First, the degraded image to be restored is received through the input image module; then, the degraded image to be restored is input into the image patch embedding module, which performs local feature extraction and feature mapping on the degraded image to be restored to obtain shallow image features; subsequently, the shallow image features are input into the encoder, which consists of multiple encoding stages or encoding modules. Each encoding stage or encoding module in the encoder uses one or more dual-pathway representation modules to perform complementary representation of the image structure information and detail information in the shallow image features, and extracts encoding features at different spatial scales step by step between adjacent encoding stages or encoding modules through downsampling; after encoding is completed, the deep encoding features of the encoder are used as the initial input of the decoder. The decoder includes multiple dual-pathway representation modules and multiple feature fusion modules cascaded in sequence. The feature fusion module is set between two adjacent dual-pathways. Specifically, the initial features received by the decoder are first input to the first dual-path representation module. The features output by this dual-path representation module are upsampled and then sent to the corresponding feature fusion module. The features output by the feature fusion module are used as the input to the next level dual-path representation module. This process is performed step by step until the last level dual-path representation module directly outputs the decoded features. The decoded features are further input to the refinement module, which consists of one or more dual-path representation modules. While maintaining the feature space resolution, it further enhances the structural and detail representations. Its internal processing is the same as that of the aforementioned dual-path representation modules. Finally, the refined features are mapped to the image space in the refinement module, and the predicted image mapped to the image space is globally residually fused with the input degraded image by the global residual fusion module to obtain the restored clear image. The restored clear image is then output by the output image module.

[0068] It should be noted that this embodiment is only a brief illustrative description of the overall process of an image restoration method. Detailed descriptions of each step can be found in the relevant content of the foregoing embodiments, and will not be repeated here. It is understood that this application does not impose any limitations on this.

[0069] In summary, the key technical points of the image restoration method provided in this application are: (1) Dual-path feature processing mechanism: Parallel structural path and detail path are set for the same degraded image features to extract global structural information and local texture details of the image respectively, and the two types of features are fused complementaryly to reduce the mutual interference of different types of image information in the same processing path.

[0070] (2) Frequency-aware detail extraction and feature optimization mechanism: Multi-scale frequency decomposition, frequency band modulation and frequency mixing are used to extract image details, and local frequency transformation is used to optimize the fused features to enhance effective frequency information and reduce degradation interference.

[0071] Compared to image restoration methods that employ shared feature representations, the image restoration method provided in this application processes global structural information and local detail information through structural and detail paths respectively, reducing mutual interference between different types of image information in the same processing path. Furthermore, this application also processes different frequency components in the image through multi-scale frequency decomposition and local frequency-aware optimization, enhancing effective frequency information and reducing degradation interference. Therefore, this application can simultaneously restore the overall structure of objects and recover texture details, and can be used for the restoration of noisy, blurred, foggy, rainy, snowy, low-light, and composite degraded images.

[0072] The image restoration method provided in this application can be applied to hardware and software products with image quality enhancement requirements, including but not limited to: smartphones, digital cameras, video surveillance equipment, medical imaging equipment, remote sensing image processing systems, industrial vision inspection equipment, autonomous driving vision systems, and image processing software. The image restoration method provided in this application can serve as an image restoration module in the aforementioned application products, used to process images with noise, blur, fog, rain, snow, low light, and multiple simultaneous degradations.

[0073] In addition, this application also provides alternative solutions, the details of which are as follows: (1) Alternatives to feature extraction methods: Structural pathways can use convolution, attention mechanisms, state-space models or multi-scale networks; detail pathways can use Gaussian difference, wavelet transform, Laplace pyramid, discrete cosine transform or other frequency decomposition methods.

[0074] (2) Replacement of feature fusion and frequency optimization methods: The features of the structural path and the detail path can be fused by addition, weighting, splicing, gating or attention; the local fast Fourier transform in the frequency-aware refinement module can also be replaced by global or local discrete cosine transform, wavelet transform and other frequency domain processing methods.

[0075] (3) Replacement of network structure and application form: The overall network can adopt convolutional neural network, Transformer network, state space network or combination thereof; the dual-path representation module can be set in all or part of the network stage, and can be extended from single image restoration to video image restoration.

[0076] Please see Figure 7 This application also provides an image restoration device 700 that can implement the above-described method. The device includes the following modules: The feature embedding processing module 701 is used to input the degraded image to be restored into the image block embedding module for feature embedding processing to obtain shallow image features; The feature encoding processing module 702 is used to input the shallow image features into the encoder for feature encoding processing to obtain multi-scale encoded features; wherein, the encoder is composed of several encoding modules, and the encoding module is composed of a dual-path representation module, the dual-path representation module includes a structural path and a detail path, the structural path is used to extract the structural information of the image, and the detail path is used to extract the detail information of the image; The feature decoding processing module 703 is used to input the multi-scale encoded features into the decoder for feature decoding processing to obtain the target decoded features; The feature refinement processing module 704 is used to input the target decoded features into the refinement module for feature refinement processing to obtain refined features, and to map the refined features to the image space to obtain the predicted image; The global residual fusion module 705 is used to input the predicted image and the degraded image to be restored into the global residual fusion module for global residual fusion processing to obtain the target restored image.

[0077] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0078] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0079] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0080] Please see Figure 8 , Figure 8 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 801 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 802 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and is called and executed by the processor 801 using the methods described in the embodiments of this application. The 803 input / output interface is used to implement information input and output. The communication interface 804 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 805 transmits information between various components of the device (e.g., processor 801, memory 802, input / output interface 803, and communication interface 804); The processor 801, memory 802, input / output interface 803, and communication interface 804 are connected to each other within the device via bus 805.

[0081] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0082] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0083] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0084] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0085] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0086] This application provides an image restoration method and related apparatus. It extracts complete shallow image features through image patch embedding, providing reliable basic feature support for subsequent feature encoding and image restoration. An encoder containing a dual-path representation module performs multi-scale encoding of shallow features, independently extracting global structural information and local detail information using structural and detail paths respectively, reducing mutual interference between different types of image information in the same processing path. A decoder completes the decoding and reconstruction of multi-scale features, ensuring the complete restoration of multi-scale feature information. A refinement module further optimizes feature representation capabilities, improving image prediction quality. Global residual fusion fully combines the original degraded image information with the predicted image information, compensating for information loss during feature restoration. Therefore, this application can simultaneously restore the overall structure of the object and the texture details, thereby improving the structural and detail restoration capabilities of degraded images and effectively enhancing the restoration accuracy of degraded images.

[0087] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0088] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0089] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0090] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0091] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0092] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0093] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0094] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0095] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0096] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0097] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. An image restoration method, characterized in that, The method includes the following steps: The degraded image to be restored is input into the image block embedding module for feature embedding processing to obtain shallow image features; The shallow image features are input into an encoder for feature encoding processing to obtain multi-scale encoded features; wherein, the encoder consists of several encoding modules, and each encoding module consists of a dual-path representation module, which includes a structural path and a detail path. The structural path is used to extract structural information of the image, and the detail path is used to extract detailed information of the image. The multi-scale encoded features are input into the decoder for feature decoding processing to obtain the target decoded features; The target decoding features are input into the refinement module for feature refinement processing to obtain refined features, and the refined features are mapped to the image space to obtain the predicted image; The predicted image and the degraded image to be restored are input into the global residual fusion module for global residual fusion processing to obtain the target restored image.

2. The method according to claim 1, characterized in that, The step of inputting the shallow image features into the encoder for feature encoding processing to obtain multi-scale encoded features includes: The shallow image features are input into the encoder, and the first encoding module in the encoder is determined as the current encoding module, and the shallow image features are determined as the current image input features; The dual-path representation module in the current encoding module performs complementary representation of the image structure information and image detail information in the current image input features to obtain the current intermediate representation result. If the current encoding module is not the last encoding module in the encoder, then the next encoding module in the encoder is determined as the current encoding module, the current intermediate representation result is determined as the current image input feature, and the process of performing complementary representation on the image structure information and image detail information in the current image input feature through the dual-path representation module in the current encoding module to obtain the current intermediate representation result is repeated until the current encoding module is the last encoding module in the encoder, and the current intermediate representation result output by the last encoding module is determined as the deep encoding feature of the encoder. The multi-scale coding features are constructed based on the current intermediate representation results output by each of the coding modules.

3. The method according to claim 2, characterized in that, The step of performing complementary representation of image structure information and image detail information in the current image input features through the dual-path representation module in the current encoding module to obtain the current intermediate representation result includes: The current image input features are normalized through the first normalization layer in the dual-path representation module to obtain the first normalized features; The structural path features are obtained by performing feature transformation operations on the first normalized features through the structural path in the dual-path characterization module. By using the detail path in the dual-path characterization module, feature transformation operation is performed on the first normalized feature to obtain the detail path feature; The dual-path fusion unit in the dual-path representation module performs element-wise residual addition on the structural path features, the detail path features, and the current image input features to obtain the dual-path fusion features. The dual-path fusion features are normalized through the second normalization layer in the dual-path representation module to obtain the second normalized features. The frequency-aware refinement module in the dual-path characterization module refines the second normalized feature to obtain the frequency-aware refined feature. The current intermediate representation result is obtained by adding the element-wise residuals of the frequency-aware refinement feature and the dual-path fusion feature.

4. The method according to claim 3, characterized in that, The step of performing feature transformation on the first normalized feature through the structural pathway in the dual-pathway representation module to obtain structural pathway features includes: The first normalized feature is subjected to feature projection processing through the input convolutional unit in the structural path to obtain the structural projection feature. Local spatial information is extracted from the structural projection features by using the deep convolutional units in the structural pathway to obtain structural spatial aggregation features; The structural spatial aggregation features are subjected to complementary feature partitioning to obtain a first feature group and a second feature group; The first feature group and the second feature group are exchanged through the bidirectional feature interaction unit in the structural path to obtain the first interactive feature and the second interactive feature. By performing element-by-element interaction on the first interactive feature and the second interactive feature, a bidirectional interactive fusion feature is obtained; The output convolutional units in the structural pathway are used to perform channel mapping and feature aggregation on the bidirectional interactive fusion features to obtain the structural pathway features.

5. The method according to claim 3, characterized in that, The step of performing feature transformation on the first normalized feature through the detail path in the dual-path representation module to obtain detail path features includes: The first normalized feature is processed by channel mapping through the input projection unit in the detail path to obtain the detail projection feature. The Gaussian difference frequency refinement unit in the detail path performs Gaussian difference frequency decomposition on the detail projection features to obtain frequency features. The frequency detail features are obtained by generating adaptive gating information based on the detail projection features through the frequency band gating unit in the detail path, and then modulating the frequency features element by element based on the adaptive gating information. The global context extraction unit in the detail path performs global spatial information aggregation on the detail projection features to obtain global context features. The global context features and the frequency detail features are spliced ​​together by the feature splicing and fusion unit in the detail path to obtain the fused detail features; The fused detail features are obtained by channel mapping and feature aggregation through the output projection unit in the detail path.

6. The method according to claim 3, characterized in that, The frequency-aware refinement module in the dual-path representation module refines the second normalized feature to obtain the frequency-aware refined feature, including: The second normalized feature is processed by the input projection unit in the frequency sensing refinement module to obtain the interactive projection feature. The interactive projection features are subjected to feature splitting processing to obtain a first feature branch and a second feature branch; The first feature branch is locally extracted by using the deep convolutional unit in the frequency-aware thinning module to obtain the locally extracted features. The local extracted features are nonlinearly activated by the nonlinear activation unit in the frequency sensing refinement module to generate adaptive modulation information. The adaptive modulation information and the feature response of the second feature branch are multiplied element-wise by the element-wise multiplication unit in the frequency sensing refinement module to obtain the gated enhancement feature. The output projection unit in the frequency sensing refinement module performs channel mapping and feature aggregation on the gated enhancement features to obtain mapped and aggregated features. The frequency-aware refinement feature is obtained by refining the mapped aggregated feature in the local frequency domain through the local image block frequency domain processing unit in the frequency-aware refinement module.

7. An image restoration device, characterized in that, The device includes the following modules: The feature embedding processing module is used to input the degraded image to be restored into the image block embedding module for feature embedding processing to obtain shallow image features; The feature encoding processing module is used to input the shallow image features into the encoder for feature encoding processing to obtain multi-scale encoded features; wherein, the encoder is composed of several encoding modules, and each encoding module is composed of a dual-path representation module, which includes a structural path and a detail path. The structural path is used to extract the structural information of the image, and the detail path is used to extract the detail information of the image. The feature decoding processing module is used to input the multi-scale encoded features into the decoder for feature decoding processing to obtain the target decoded features; The feature refinement module is used to input the target decoded features into the refinement module for feature refinement processing to obtain refined features, and to map the refined features to the image space to obtain the predicted image; The global residual fusion module is used to input the predicted image and the degraded image to be restored into the global residual fusion module for global residual fusion processing to obtain the target restored image.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images