Image restoration method based on frequency domain decoupling, computer device and storage medium

Abstract

The present application discloses an image restoration method, computer device, and storage medium based on frequency domain decoupling. The method comprises: obtaining an image to be processed and inputting the image to be processed into a decoupling network; performing multi-scale transformation processing on the image to be processed to obtain a multi-scale image set; inputting the multi-scale image set into a multi-scale amplitude restoration module for amplitude restoration processing to obtain a restored and brightened image set; performing component extraction processing on the restored and brightened image set and the image to be processed to obtain prior information; and inputting the image to be processed and the prior information into a Fourier-based decoupling module for image restoration processing to obtain a restored image. The embodiments of the present application can perform decoupling processing based on different degradation features using a Fourier-based decoupling module, thereby improving image restoration performance and can be widely applied in the field of image processing technology.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to an image restoration method based on frequency domain decoupling, a computer device and a storage medium. BACKGROUND

[0002] In the related art, there is an image restoration method of dark light enhancement and deblurring, that is, image restoration is performed on a specific and single degradation, or an intuitive cascading method is used for image restoration. However, in actual application, it is found that these methods cannot effectively decouple dark light degradation and blurring degradation, cannot effectively model different degradations, and cannot restore clear edges in dark light blurred images, thereby affecting the effect of image restoration.

[0003] To sum up, the technical problems existing in the related art need to be improved. SUMMARY

[0004] The main purpose of the embodiments of the present application is to provide an image restoration method based on frequency domain decoupling, a computer device and a storage medium, which can improve the restoration ability of clear edges of images.

[0005] To achieve the above purpose, one aspect of the embodiments of the present application provides an image restoration method based on frequency domain decoupling, which comprises:

[0006] Obtaining a to-be-processed image, inputting the to-be-processed image into a decoupling network, the decoupling network comprising a multi-scale amplitude restoration module and a Fourier-based decoupling module;

[0007] Performing multi-scale transformation processing on the to-be-processed image to obtain a multi-scale image set;

[0008] Inputting the multi-scale image set into the multi-scale amplitude restoration module for amplitude restoration processing to obtain a restored brightened image set;

[0009] Performing component extraction processing on the restored brightened image set and the to-be-processed image to obtain prior information;

[0010] Inputting the to-be-processed image and the prior information into the Fourier-based decoupling module for image restoration processing to obtain a restored image.

[0011] In some embodiments, the step of inputting the multi-scale image set into the multi-scale amplitude restoration module for amplitude restoration processing to obtain a restored brightened image set comprises the following steps:

[0012] Obtaining a scaling factor;

[0013] Performing gamma correction processing on the multi-scale image set according to the scaling factor to obtain the restored brightened image set.

[0014] In some embodiments, the component extraction processing of the restored brightened image set and the to-be-processed image to obtain prior information comprises the following steps:

[0015] The amplitude component extraction processing is performed on each image in the restored brightened image set to obtain an amplitude component set;

[0016] The phase component extraction processing is performed on the to-be-processed image to obtain a phase component;

[0017] The amplitude component set and the phase component are determined as the prior information.

[0018] In some embodiments, the Fourier-based decoupling module comprises a decoupling self-attention layer, a decoupling feedforward layer, and a cross-attention feedforward network, and the inputting of the to-be-processed image and the prior information into the Fourier-based decoupling module for image restoration processing to obtain a restored image comprises the following steps:

[0019] The feature self-attention calculation processing is performed on the to-be-processed image based on the decoupling self-attention layer to obtain a first feature vector;

[0020] The frequency learning and spatial learning processing is performed on the first feature vector based on the decoupling feedforward layer to obtain a second feature vector;

[0021] The cross-attention calculation processing is performed on the second feature vector and the prior information based on the cross-attention feedforward network to obtain the third feature vector;

[0022] The convolution processing is performed on the third feature vector to obtain a feature map, and the feature map is added to the to-be-processed image to obtain the restored image.

[0023] In some embodiments, the feature self-attention calculation processing of the to-be-processed image based on the decoupling self-attention layer to obtain a first feature vector comprises the following steps:

[0024] The linear change processing is performed on the to-be-processed image to obtain a query vector, a key vector, a value vector, and an information vector, and the information vector comprises amplitude information and phase information;

[0025] The phase correlation calculation processing is performed on the query vector and the key vector according to the information vector to obtain phase-related attention;

[0026] The amplitude self-attention calculation processing is performed on the query vector and the key vector according to the information vector to obtain amplitude modulation attention;

[0027] The query vector and the key vector are subjected to mixed attention calculation processing to obtain mixed attention;

[0028] The value vector is subjected to weighted calculation processing according to the phase-related attention, the amplitude modulation attention and the mixed attention to obtain the first feature vector.

[0029] In some embodiments, the decoupling feedforward layer includes a frequency branch and a spatial branch, and the first feature vector is subjected to frequency learning and spatial learning processing based on the decoupling feedforward layer to obtain a second feature vector, including the following steps:

[0030] The amplitude and phase of the first feature vector are subjected to frequency selection and offset processing based on the frequency branch to obtain a frequency feature;

[0031] The first feature vector is subjected to local feature learning processing in the spatial domain based on the spatial branch to obtain a spatial feature;

[0032] The frequency feature and the spatial feature are subjected to aggregation and nonlinear transformation processing to obtain the second feature vector.

[0033] In some embodiments, the second feature vector and the prior information are subjected to cross-attention calculation processing based on the cross-attention feedforward network to obtain the third feature vector, including the following steps:

[0034] The second feature vector and the prior information are subjected to Fourier cross-attention calculation processing to obtain an intermediate feature vector;

[0035] The intermediate feature vector is subjected to spatial modulation processing to obtain the third feature vector.

[0036] In some embodiments, before the to-be-processed image is input into the decoupling network, the method further includes training the decoupling network, including the following steps:

[0037] Obtaining a training image;

[0038] The training image is input into the multi-scale amplitude restoration module for coarse alignment loss and amplitude loss calculation processing to obtain a first loss value;

[0039] The parameters of the multi-scale amplitude restoration module are adjusted according to the first loss value, and the adjusted parameters are frozen;

[0040] The training image is input into the Fourier-based decoupling module for pixel loss, frequency loss and perception loss calculation processing to obtain a second loss value;

[0041] According to the second loss value, a parameter of the Fourier-based decoupling module is adjusted to obtain a trained decoupling network.

[0042] To achieve the above object, another aspect of the embodiments of the present application provides an image restoration system based on frequency domain decoupling, which comprises:

[0043] A first module is configured to acquire a to-be-processed image and input the to-be-processed image into a decoupling network, wherein the decoupling network comprises a multi-scale amplitude restoration module and a Fourier-based decoupling module.

[0044] A second module is configured to perform multi-scale transformation on the to-be-processed image to obtain a multi-scale image set.

[0045] A third module is configured to input the multi-scale image set into the multi-scale amplitude restoration module to perform amplitude restoration to obtain a restored brightened image set.

[0046] A fourth module is configured to perform component extraction on the restored brightened image set and the to-be-processed image to obtain prior information.

[0047] A fifth module is configured to input the to-be-processed image and the prior information into the Fourier-based decoupling module to perform image restoration to obtain a restored image.

[0048] To achieve the above object, another aspect of the embodiments of the present application provides a computer device, which comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the above method when executing the computer program.

[0049] To achieve the above object, another aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above method.

[0050] The embodiments of the present application have at least the following advantages: the embodiments of the present application provide an image restoration method based on frequency domain decoupling, a computer device and a storage medium, which input a multi-scale image set into a multi-scale amplitude restoration module to perform amplitude restoration to obtain a restored brightened image set, can adjust the brightness of an image by restoring the amplitude, and input a to-be-processed image and prior information into a Fourier-based decoupling module to perform image restoration to obtain a restored image, can perform decoupling processing on different degradation characteristics based on the Fourier-based decoupling module, and improve the image restoration effect. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1is a flowchart of an image restoration method based on frequency domain decoupling provided by an embodiment of the present application;

[0052] Figure 2 (a) is a normal light clear image, Figure 2 (b) is an amplitude map corresponding to the normal light clear image, Figure 2 (c) is a phase map corresponding to the normal light clear image, Figure 2 (d) is a normal light blurred image after exchanging the phase, Figure 2 (e) is Figure 2 (d) is a local magnified image corresponding to the red box, Figure 2 (f) is an image obtained by performing histogram equalization processing on Figure 2 (a), Figure 2 (g) is Figure 2 (b) is a spatial domain representation of Figure 2 (h) is a dark light blurred image, Figure 2 (i) is an amplitude map corresponding to the dark light blurred image, Figure 2 (j) is a phase map corresponding to the dark light blurred image, Figure 2 (k) is a dark light clear image after exchanging the phase, Figure 2 (l) is Figure 2 (k) is a local magnified image corresponding to the red box, Figure 2 (m) is an image obtained by performing histogram equalization processing on Figure 2 (h), Figure 2 (n) is Figure 2 (b) is a spatial domain representation of

[0053] Figure 3 is a schematic diagram of the overall structure of a decoupling network provided by an embodiment of the present application;

[0054] Figure 4 is a schematic diagram of the structure of a multi-scale amplitude restoration module provided by an embodiment of the present application;

[0055] Figure 5 is a schematic diagram of the structure of a decoupling self-attention layer provided by an embodiment of the present application;

[0056] Figure 6 is a schematic diagram of the structure of a decoupling feedforward layer provided by an embodiment of the present application;

[0057] Figure 7 is a schematic diagram of the structure of a cross-attention feedforward network provided by an embodiment of the present application;

[0058] Figure 8 is a schematic diagram of the hardware structure of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purposes, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application. When the following description refers to the accompanying drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with embodiments of the present application. They are only examples of apparatuses and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0060] It can be understood that the terms "first", "second" and the like used in the present application can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "when" or "in response to determining".

[0061] The terms "at least one", "multiple", "each", "any" and the like used in the present application include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as understood by those skilled in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0063] In the related art, there are image restoration methods for dark light enhancement and deblurring, i.e. image restoration on specific and single degradation, or intuitive cascading method for image restoration. However, in practical applications, it is found that these methods cannot effectively decouple dark light degradation and blurring degradation, cannot effectively model different degradations, resulting in the inability to restore clear edges in dark light blurred images, affecting the effect of image restoration. There are also methods in the related art that can simultaneously process two degradations, but due to the poor effect caused by the failure of these methods to effectively model different degradations, the effect of image restoration is also affected.

[0064] Exemplarily, for example, the image restoration method can adopt a dark map enhancement method. Common methods of dark map enhancement based on traditional image processing techniques include methods based on histogram equalization, methods based on retinex theory, and methods based on gamma correction. However, these traditional dark light enhancement algorithms are heavily dependent on manual prior knowledge and show poor generalization ability and significant color bias when faced with diverse scenes. In addition, these methods are often sensitive to noise and can easily amplify noise in the image, thereby affecting the dark light enhancement effect. Dark light enhancement methods based on deep learning have better enhancement effect and can be roughly divided into time domain-based methods and frequency domain-based methods. The time domain-based method directly processes the pixel values of the image, that is, the image is enhanced in the spatial domain. Existing methods based on the spatial domain often ignore Fourier frequency information, which can capture global attributes and has the ability to decouple image degradation and content components. These methods mainly focus on the influence of amplitude on brightness. However, they often ignore the influence of phase, especially in scenes with blur. Therefore, this type of method often cannot well restore the blur degradation in the image when there is both dark light and blur degradation.

[0065] Exemplarily, the image restoration method can adopt a deblurring method. Traditional image deblurring methods mainly include blind deblurring and non-blind deblurring methods. Blind deblurring methods aim to estimate the blur kernel and the deblurred image simultaneously. However, due to the high ill-posedness of the blind deblurring problem, these methods are usually sensitive to initial conditions and are prone to local optimization, resulting in unsatisfactory deblurring effect. Non-blind deblurring methods assume that the blur kernel is known and restore the original image through inverse convolution technology. However, in practical applications, it is often difficult to accurately obtain the blur kernel, so the actual effect of these methods is often severely affected by the error in the estimation of the blur kernel. Deep learning-based deblurring methods have stronger learning ability and can adaptively learn the blur degradation information. However, these methods ignore the possible simultaneous existence of dark light degradation in actual scenes. In a dark light environment, problems such as pixel saturation noise in high brightness areas are very unfavorable for deblurring.

[0066] Exemplarily, the image restoration method can also employ a joint dark-light enhancement and deblurring method. Most current researches on image restoration mainly focus on specific and single degradation. However, in actual scenes, images are usually affected by multiple degradations. Simply employing an intuitive cascading method can lead to poor results, resulting in error accumulation and increased computational burden. The basic assumption of the joint dark-light enhancement and deblurring method is that the down-sampled blurred image can be approximated as the down-sampled clear image. However, due to the spatial domain dependency of dark-light and blur degradation, different degradations are difficult to effectively decouple, thereby limiting the performance of the designed module and having limitations in restoring sharp edges. Therefore, in the related art, the methods of separate dark-light enhancement, separate deblurring and joint dark-light enhancement and deblurring cannot effectively decouple dark-light degradation and blur degradation, thereby failing to restore clear edges in dark-light blurred images.

[0067] Therefore, in the embodiments of the present application, a kind of image restoration method based on frequency domain decoupling, computer device and storage medium are provided, which decouples dark-light degradation and blur degradation in Fourier domain, can model the physical degradation process of dark-light degradation and blur degradation based on amplitude modulation and phase correlation respectively, and through Fourier-based decoupling network for joint dark-light enhancement and deblurring, dark-light degradation and blur degradation can be effectively decoupled and modeled, thereby significantly improving the restoration ability of clear edges of image.

[0068] The image restoration method based on frequency domain decoupling provided in the embodiments of the present application relates to the technical field of image processing. The image restoration method based on frequency domain decoupling provided in the embodiments of the present application can be applied in a terminal, can be applied in a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, etc., but is not limited thereto; the server end can be configured as a standalone physical server, can be configured as a server cluster or a distributed system composed of multiple physical servers, can be configured as a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligence platforms, and the server can also be a node server in a blockchain network; the software can be an application that implements an image restoration method based on frequency domain decoupling, etc., but is not limited to the above forms.

[0069] The application is operable in a multitude of general or special computer system environments or configurations. Examples of well known computing systems, environments, and / or configurations that can be suitable for use with the application include, but are not limited to, personal computers, server computers, handheld or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media including memory storage devices.

[0070] Figure 1 is an optional flowchart of an image restoration method based on frequency domain decoupling provided by an embodiment of the application, Figure 1 The method in the above embodiment can include, but is not limited to, steps S101 to S105.

[0071] In step S101, a to-be-processed image is acquired, and the to-be-processed image is input to a decoupling network, where the decoupling network includes a multi-scale amplitude restoration module and a Fourier-based decoupling module.

[0072] In step S102, multi-scale transformation processing is performed on the to-be-processed image to obtain a multi-scale image set.

[0073] In step S103, the multi-scale image set is input to the multi-scale amplitude restoration module for amplitude restoration processing to obtain a restored brightened image set.

[0074] In step S104, component extraction processing is performed on the restored brightened image set and the to-be-processed image to obtain prior information.

[0075] In step S105, the to-be-processed image and the prior information are input to the Fourier-based decoupling module for image restoration processing to obtain a restored image.

[0076] The steps S101 to S105 shown in the above embodiment can perform dark light enhancement and deblurring processing on the to-be-processed image through the Fourier-based decoupling network. In the above embodiment, it is found that dark light degradation and blurring degradation can be reflected in the amplitude and phase components of the image, respectively. Further analysis of the two kinds of degradation shows that dark light degradation has different degradation forms in different frequency bands of the amplitude, and blurring degradation can be modeled through phase correlation. Please refer to Figure 2(a) and Figure 2 (h) are the normal-light sharp image and the dark-light blurred image, respectively, Figure 2 (b), Figure 2 (c), Figure 2 (i) and Figure 2 (j) are their corresponding amplitude and phase images, respectively. By swapping their phase, the present embodiments obtain a dark-light sharp image Figure 2 (k) and a normal-light blurred image Figure 2 (d). Figure 2 (f) and Figure 2 (m) are the histogram equalization processed versions of Figure 2 (c) and Figure 2 (j) in spatial domain. It is further demonstrated that the phase can represent the complete structure information. In addition, Figure 2 (g) and Figure 2 (n) are the spatial domain representations of Figure 2 (b) and Figure 2 (i). It can be observed that Figure 2 (g) is brighter than Figure 3 (n), but they do not reveal the structure information. This indicates that the phase mainly encodes the structure information, while the amplitude mainly encodes the brightness information. Further, by swapping the specific frequency bands of the amplitude and phase in the image, the present embodiments further explore the characteristics of different frequency bands in the amplitude and phase. When the swapped image has a high proportion of the phase of image B, the image obviously retains the structure of B and has the detail information of B, while the information of A is embodied as a blurred outline. As the proportion of the phase B decreases, the structure of A gradually becomes clear and complete, while the contribution of B decreases to the edge information. This indicates that the high-frequency phase mainly encodes the edge information, while the low-frequency phase mainly encodes the overall structure of the image. When the swapped image has a low proportion of the amplitude of image B, the image shows a similar overall brightness to image B, but has a lower contrast. As the proportion of the amplitude of image B in the swapped image increases, the color and contrast of the detail area gradually increase, approaching the effect of image B. This indicates that the low-frequency amplitude mainly encodes the overall brightness and contrast of the image, while the high-frequency amplitude encodes the color information of the detail area.

[0077] Therefore, an embodiment of the present application constructs a decoupling network based on the transformers architecture, which can perform restoration processing on an image through joint dark light enhancement and deblurring of the decoupling network. The decoupling network includes a multi-scale amplitude restoration module, which can consider the demand for personalized restoration by adjusting the coarse amplitude of the image to customize the brightness, and use the restoration result as the prior of the Fourier-based decoupling module through the Fourier-based cross-attention mechanism. The Fourier-based decoupling module can learn different representations of different degradations through the Fourier domain self-attention mechanism. In addition, considering that different frequency ranges contain unique information in terms of amplitude and phase, a filtering mechanism is introduced to amplify or attenuate specific frequency bands of features. The embodiment of the present application can effectively decouple and model dark light degradation and blur degradation, thereby significantly improving the restoration capability of clear edges of an image.

[0078] In step S101 of some embodiments, the to-be-processed image can be obtained by user uploading, or can be obtained by other means, such as from a database, without limitation. The to-be-processed image is an image that needs to be processed for dark light enhancement and deblurring, and then the to-be-processed image is input to the Fourier-based decoupling network. Please refer to Figure 4 , the decoupling network is composed of a multi-scale amplitude restoration module (MAR) and a Fourier-based decoupling module (FDformer), wherein the Fourier-based decoupling module is composed of a Fourier-based decoupling self-attention layer (FDSA), a Fourier-based decoupling feedforward layer (FDFFN), and a Fourier-based cross-attention feedforward network (FCAFFN).

[0079] In step S102 of some embodiments, the to-be-processed image is subjected to multi-scale transformation processing to obtain a multi-scale image set.

[0080] In an embodiment of the present application, by performing multi-scale transformation on the to-be-processed image, images at different scales are obtained, for example, by performing scaling processing on the to-be-processed image, a plurality of images at different scales can be obtained, and the to-be-processed image and other scale images are used as a multi-scale image set. These images are used to input the multi-scale amplitude restoration module for amplitude restoration processing, and the restored results are used as prior knowledge to guide the Fourier-based decoupling module for brightness restoration, thereby improving the effect of image restoration.

[0081] In step S103 of some embodiments, the multi-scale image set is input to the multi-scale amplitude restoration module for amplitude restoration processing to obtain a restored brightening image set.

[0082] In the embodiment of the present application, the multi-scale image set processed by the multi-scale transformation in the above steps is input into the multi-scale amplitude restoration module for amplitude restoration processing, so that the multi-scale amplitude restoration module can perform rough restoration amplitude, and the rough amplitude obtained by the restoration can be used as the brightness prior, thereby guiding the Fourier-based decoupling module to focus on the restoration details and improving the image restoration effect.

[0083] In step S103 of some embodiments, the multi-scale image set is input into the multi-scale amplitude restoration module for amplitude restoration processing to obtain a restored brightening image set, including the following steps:

[0084] Obtaining a scaling factor;

[0085] According to the scaling factor, the multi-scale image set is subjected to gamma correction processing to obtain the restored brightening image set.

[0086] In the embodiment of the present application, the scaling factor represents the ratio of the global brightness of the original input image to the global brightness expected by the user, which is used to adjust the intermediate features of the image, and can adjust the amplitude component of the image according to the scaling factor to adjust the brightness of the image, thereby realizing the brightness of the user-defined image. Specifically, please refer to Figure 5 The multi-scale amplitude restoration module adopts the U-Net architecture, uses the FP block and the convolution layer to extract features, and processes each image in the multi-scale image set. For each scale, the embodiment of the present application sets a pixel-level correction factor γ n with a value range of (0, 1). The gamma correction is applied to the input image to restore the rough brightness and obtain the restored brightening image. The expression of the restored brightening image is as follows:

[0087]

[0088] In the formula, I n represents the image of the nth scale, γ n represents the pixel-level correction factor, represents the restored brightening image corresponding to the nth scale. The brightness information can be obtained by extracting the amplitude component of the restored brightening image. In addition, for a dark light blur image, the brightness of the corresponding restoration result is not unique. Therefore, the embodiment of the present application encodes the custom brightness in the rough amplitude. Specifically, for a gray-scale image f(x, y) with a scale of HxW, the amplitude component |F(0, 0)| is represented as that is, the average value of all pixel values in the image f(x, y). The amplitude component |F(0, 0)| is regarded as the global brightness of the image f(x, y) Similarly, for an RGB image, the global brightness can be calculated after being converted into a gray-scale image.

[0089] To study the influence of the amplitude component |F(0,0)| on the whole image f(x,y), a new image f'(x,y) is generated by amplifying |F(0,0)| by a coefficient a:

[0090]

[0091] where H represents the height of the image, W represents the width of the image, and F(u,v) represents the Fourier transform of the image f(x,y). This operation is equivalent to increasing each pixel value of the image in the spatial domain without considering the color distribution, resulting in color distortion. Therefore, the embodiments of the present application consider using an implicit method to customize the brightness, introducing a scaling factor representing the ratio of the global brightness of the original input image to the global brightness desired by the user to adjust the intermediate features and pass through the gamma correction, so as to be able to achieve user-defined brightness in the amplitude component of the restored brightened image.

[0092] In step S104 of some embodiments, the component extraction processing is performed on the set of restored brightened images and the to-be-processed image to obtain prior information, including the following steps:

[0093] The amplitude component extraction processing is performed on each image in the set of restored brightened images to obtain a set of amplitude components;

[0094] The phase component extraction processing is performed on the to-be-processed image to obtain a phase component;

[0095] The set of amplitude components and the phase component are determined as the prior information.

[0096] In the embodiments of the present application, the amplitude component of the restored brightened image and the phase component of the original dark light blurred image are extracted respectively, and the set of amplitude components and the phase component are determined as the prior information, so that the extracted amplitude component and phase component can be input as prior information into the Fourier-based cross-attention feedforward network to guide the decoupling module to perform image restoration, thereby improving the processing efficiency of image restoration.

[0097] In step S105 of some embodiments, the to-be-processed image and the prior information are input into the Fourier-based decoupling module to perform image restoration processing to obtain a restored image, including the following steps:

[0098] The feature self-attention calculation processing is performed on the to-be-processed image based on the decoupling self-attention layer to obtain a first feature vector;

[0099] The frequency selection and offset learning processing is performed on the first feature vector in a high-dimensional space based on the decoupling feedforward layer to obtain a second feature vector;

[0100] perform cross-attention calculation processing on the second feature vector and the prior information based on the cross-attention feedforward network to obtain a third feature vector;

[0101] perform convolution processing on the third feature vector to obtain a feature map, and perform addition processing on the feature map and the to-be-processed image to obtain the restored image.

[0102] In the embodiment of the present application, the Fourier-based decoupling module includes a decoupling self-attention layer, a decoupling feedforward layer, and a cross-attention feedforward network. The decoupling self-attention layer is used to perform feature self-attention calculation processing on the to-be-processed image, so that different representations of blur degradation and dark degradation can be obtained through self-attention, and a first feature vector is obtained. Since dark degradation and blur degradation have different representations in different frequency bands, the Fourier decoupling feedforward layer is used to process the first feature vector, so that frequency selection and offset in a high-dimensional space can be adaptively learned. In addition, the introduction of image multi-scale features can improve the effect of image deblurring. The Fourier cross-attention feedforward layer (FCAFFN) with a Fourier cross-attention (FCA) mechanism is used to perform cross-attention calculation on multi-scale phase features and restored rough amplitude features, and finally a restored image is output, which can effectively decouple and model dark degradation and blur degradation, thereby significantly improving the restoration capability of clear edges of the image.

[0103] In some embodiments, the feature self-attention calculation processing on the to-be-processed image based on the decoupling self-attention layer to obtain a first feature vector includes the following steps:

[0104] performing linear transformation processing on the high-dimensional features of the to-be-processed image to obtain a query vector, a key vector, a value vector, and an information vector, the information vector including amplitude information and phase information;

[0105] performing phase correlation calculation processing on the query vector and the key vector according to the information vector to obtain phase-related attention;

[0106] performing amplitude self-attention calculation processing on the query vector and the key vector according to the information vector to obtain amplitude modulation attention;

[0107] performing hybrid attention calculation processing on the query vector and the key vector to obtain hybrid attention;

[0108] performing weighted calculation processing on the value vector according to the phase-related attention, the amplitude modulation attention, and the hybrid attention to obtain the first feature vector.

[0109] In the embodiments of the present application, the decoupled self-attention layer is used to obtain different representations of blur degradation and dark degradation through self-attention. Please refer to Figure 6 After the decoupled self-attention layer normalizes the image to be processed, linear transformation is performed through 1*1 convolution and 3*3 depth separable convolution, and the query vector, key vector, value vector and information vector can be obtained, wherein the information vector provides additional amplitude and phase information for phase-dependent attention and amplitude modulation attention. The phase-dependent attention is obtained by performing phase-dependent calculation on the query vector and the key vector according to the information vector. The process of calculating the phase-dependent attention is deduced and analyzed as follows. Specifically, for blur degradation, blur can be approximately represented by averaging a series of consecutive frames, and the expression is as follows:

[0110]

[0111] Among them, represents the observed blurred image, g represents the mapping of the latent clear signal to the observed camera response function, m represents the total number of frames during image shooting, and S[i] represents the clear signal of the i-th frame. The blur information is obtained by a pure phase method in the embodiments of the present application. The embodiments of the present application assume First, considering the case of global motion, the above formula can be converted to:

[0112]

[0113] Among them, △x i represents the displacement relative to the first frame, so that represents the i-th frame of the image.

[0114] Since the exposure time is usually not too long, the difference △x i+1 -△x i between adjacent frames is small, and the embodiments of the present application can assume that this process is approximately uniform linear motion, and the displacement △x i+1 -△x i between adjacent frames is consistent, that is, △x

[0115]

[0116] In combination with the Fourier shift theorem, the embodiments of the present application perform Fourier transform on the above formula:

[0117]

[0118] Among them, A i (ω) and P i(ω) represent the amplitude and phase, respectively. ΔΡ(ω) = 2π△xω represents the phase difference between adjacent frames, and Ρ1(ω) is the phase of the first frame. Here, embodiments of the present application only consider the representation of blur in the phase, so A i (ω) can be set to 1. Using the geometric series summation formula, we can get:

[0119]

[0120] Here, embodiments of the present application assume the phase of two images blur1 and blur2 with the same initial phase but different parameters m. The phase correlation between blur1 and blur2 can be represented as:

[0121]

[0122] where m1 and m1 represent the blur degree of blur1 and blur2, respectively, and there is a small difference△m between them (m1 = m2 +△m,△m→0). Therefore, embodiments of the present application can define the function G(m) = ln(e jm△P(ω) -1), and since△m→0, the formula can be converted to:

[0123]

[0124] Considering that G(m) is a monotonically increasing function of m, and mΔP(ω) represents the phase difference between the first frame and the last frame, G(m) can be regarded as a function related to the blur degree. Therefore, the phase correlation between two blurred images can reflect their relative blur degree. Embodiments of the present application integrate the phase correlation into the self-attention mechanism to explore the representation of blur information. In the Fourier domain, the dot product of the query vector and the key vector can be used as self-attention Self-attention The formula for calculating is as follows:

[0125]

[0126] where amp and phase represent the amplitude and phase components of the variable, respectively. In order to extract the relationship between the phase of and , while not changing the mathematical properties of the self-attention calculation, embodiments of the present application set and to a constant 1, and take negative. Thus, embodiments of the present application can obtain a new form of self-attention, and the formula for the first attention is as follows:

[0127]

[0128] According to the Fourier shift theorem, setting the phase to a negative value is equivalent to rotating the matrix by 180 degrees in the spatial domain, which still maintains the mathematical properties of self-attention in and weighted sum calculation. This shows that the phase correlation between and has the mathematical properties of self-attention. It can be regarded as a self-attention calculation without amplitude, which is robust to illumination changes. This property makes it possible to effectively extract blur degradation information even in dark light environments.

[0129] To extract dark light degradation information, the amplitude-based self-attention mechanism is extracted in the embodiments of the present application. The embodiments of the present application regard as the weight of This operation is similar to setting as a filter to amplify or attenuate the frequency range in , and the second self-attention calculation formula can be obtained as follows:

[0130]

[0131] Similarly, it can be found that also has the mathematical properties of self-attention calculation.

[0132] In the Fourier domain, the amplitude represents the intensity of different frequency components in the image, and a larger amplitude indicates that the corresponding frequency has more energy. The phase measures the degree of shift of each frequency wave relative to its starting position. In self-attention calculation, only the phase is involved, which is equivalent to shifting the plane wave of the same intensity and different frequencies. Although the structural information is preserved, it loses the details of the wave intensity, making it difficult to identify the key feature position. The amplitude-based self-attention calculation cannot represent effective image information because it does not contain any image-related structure. Therefore, the embodiments of the present application introduce an additional branch ε in the above self-attention calculation to supplement the missing amplitude and phase information, i.e., the information vector. Based on the information vector, the calculation formulas of the above first attention and second attention are modified, and the calculation formulas of the phase-related attention amplitude modulation attention and hybrid attention can be obtained as follows:

[0133]

[0134] wherein is independent of the amplitude of and This mechanism explores the relationship between the phase and the amplitude of the image, which is more robust to illumination changes. and between the blur information and the dark-light degradation, the features related to dark-light degradation are avoided to be introduced. Similarly, mainly explores and the amplitude features related to dark-light degradation, and avoids introducing the features related to the structure in and . then considers the dark-light degradation and the blur degradation in and at the same time. The embodiments of the present application can design special feature extraction methods for different degradation features by decoupling the self-attention layer for feature self-attention calculation and processing of the image to be processed, can effectively obtain different degradation information, and realize effective restoration of the mixed degradation image.

[0135] In some embodiments, the decoupled feedforward layer includes a frequency branch and a spatial branch, and the first feature vector is subjected to frequency selection and offset learning processing in a high-dimensional space based on the decoupled feedforward layer to obtain a second feature vector, including the following steps:

[0136] performing frequency selection and offset processing on the amplitude and phase of the first feature vector based on the frequency branch to obtain a frequency feature;

[0137] performing inter-channel local feature learning processing on the first feature vector based on the spatial branch to obtain a spatial feature;

[0138] performing aggregation and nonlinear transformation processing on the frequency feature and the spatial feature to obtain the second feature vector.

[0139] In the embodiments of the present application, since the dark-light and blur degradation have different representations in different frequency bands, the Fourier decoupled feedforward layer (FDFFN) can adaptively learn the frequency selection and offset in a high-dimensional space, please refer to Figure 7 , the Fourier decoupled feedforward layer includes a frequency branch and a spatial branch, wherein the frequency branch uses a learnable filter to perform frequency selection and offset operation on the amplitude and phase of the feature respectively. Specifically, the embodiments of the present application divide the first feature vector into p x p small blocks by patch segmentation operation, and then use a learnable filter to adjust the amplitude and phase information in the Fourier domain to obtain a frequency feature Let x n represent the input feature at scale n, and the expression of the frequency feature is as follows:

[0140]

[0141] where PM p×p () represents a patch merging operation, and PSp×p (x n ) represents a patch segmentation operation. and are learnable parameters, initialized as 2π and 1 respectively, representing no change to phase and amplitude, learning feature modulation from initial feature space.

[0142] In the spatial domain branch, two 3x3 deep convolutions and a GELU nonlinear activation function are used to realize local feature learning between channels to obtain spatial features. After the spatial features and frequency features are fused and aggregated by addition, a nonlinear transformation is introduced through a gating mechanism, followed by a 1x1 convolution to reduce the dimension to the original size to obtain a second feature vector, the expression of which is as follows:

[0143]

[0144] In the formula, represents the second feature vector, represents a 1x1 convolution, Gate represents a gating mechanism, represents spatial features, represents frequency features. By decoupling the feedforward layer, the application embodiment can adaptively learn frequency selection and offset in high-dimensional space, thereby improving the image restoration effect.

[0145] In some embodiments, the cross-attention feedforward network is based on the cross-attention calculation and processing of the second feature vector and the prior information to obtain the third feature vector, including the following steps:

[0146] Fourier cross-attention calculation and processing of the second feature vector and the prior information to obtain an intermediate feature vector;

[0147] Spatial modulation processing of the intermediate feature vector to obtain the third feature vector.

[0148] In the application embodiment, please refer to Figure 8 , the amplitude variable and the phase variable are obtained after the second feature vector is normalized and Fourier transformed, and the Fourier cross-attention calculation is performed in combination with the prior information, and the formula of the cross-attention calculation is as follows:

[0149] Normalization, 1x1 convolution used to adjust the phase and amplitude to the same dimension as the features. used to learn the relative blur information between the input image I n and the second feature vector, and to restore the brightened image is used to modulate the amplitude Herein the embodiments of the present application adopt the phase after the restoration processing by the multi-scale amplitude restoration module Because in the case of a blurred image under the lack of normal light, the restoration process of the multi-scale amplitude restoration module can cause damage to the phase information.

[0150] In addition, considering the matching problem between the amplitude and the phase, simply combining the amplitude of the normal light clear image with the phase of the dark light blurred image can produce artifacts. To assist the adaptive alignment of the amplitude and the phase, the embodiments of the present application introduce an additional spatial modulation mechanism, and the formula of the spatial modulation is as follows:

[0151]

[0152] wherein, indicates the output feature of the n-th scale. After normalization processing on the output feature, the third feature vector can be obtained. The embodiments of the present application can adaptively align the amplitude and the phase of the image feature through the cross-attention feedforward network, thereby improving the restoration effect of the image.

[0153] In some embodiments, before the inputting of the image to be processed into the decoupling network, the method further includes training the decoupling network, including the following steps:

[0154] obtaining a training image;

[0155] inputting the training image into the multi-scale amplitude restoration module for coarse alignment loss and amplitude loss calculation processing, to obtain a first loss value;

[0156] adjusting the parameters of the multi-scale amplitude restoration module according to the first loss value, and freezing the adjusted parameters;

[0157] inputting the training image into the Fourier-based decoupling module for pixel loss, frequency loss and perception loss calculation processing, to obtain a second loss value;

[0158] adjusting the parameters of the Fourier-based decoupling module according to the second loss value, to obtain a trained decoupling network.

[0159] In the embodiments of the present application, the decoupling network adopts a two-stage training process, first trains the multi-scale amplitude restoration module, then freezes the parameters thereof, and then trains the Fourier-based decoupling module. Specifically, in the training process of the multi-scale amplitude restoration module, two loss functions, coarse alignment loss L coarse and amplitude loss L ampThe coarse alignment loss is intended to provide a rough amplitude of brightness and contrast. Due to the lack of normal lighting blurred images as an intermediate control during training, the embodiment of the present application introduces a more flexible supervision method instead of pixel-by-pixel supervision. The calculation formula of the coarse alignment loss is as follows:

[0160]

[0161] in, Indicates that the multi-scale amplitude restoration module is Output at a scale. Represents the average pooling operation of downsampling by a factor of n-1, where n represents the different scales of the multi-scale amplitude restoration module. is with The output reference image of the same scale. For the input feature map (I n ), output feature map The value at each position (i, j, c) is the average value of the elements in the pooling window, and the pooling window size is 2 n-1 ×2 n-1 , the expression of the average pooling operation is as follows:

[0162]

[0163] Amplitude loss is used to provide rough amplitude restoration. This embodiment of the application introduces an additional loss function to directly supervise the amplitude. The expression of amplitude loss is as follows:

[0164]

[0165] Therefore, the overall loss function of the training phase of the multi-scale amplitude restoration module can be expressed as the first loss L stage1 , the expression of the first loss is as follows:

[0166] L stage1 =L course +λ1L amp ;

[0167] Here, λ1 is set to 0.01 to balance the two losses.

[0168] Then, the parameters of the multi-scale amplitude restoration module are adjusted according to the calculated first loss value, and the adjusted parameters are frozen. Then, the training image is input into the Fourier-based decoupling module to calculate the pixel loss, frequency loss and perceptual loss to obtain the second loss value. Among them, the loss function of the Fourier-based decoupling module consists of pixel loss, frequency loss and perceptual loss. The pixel loss L pixThe frequency loss is used to supervise the output results in the frequency domain of the Fourier spectrum, and the expression of the frequency loss L fft is as follows:

[0169]

[0170] where Real and Imag represent the real part and the imaginary part in the Fourier domain, respectively, represents I out , and represents the two-dimensional discrete Fourier transform of I gt .

[0171] The perceptual loss is used to supervise the output results by a pre-trained VGG19 network Φ to improve the perceptual quality, and the expression of the perceptual loss L per is as follows:

[0172] L per =||Φ(I out )-Φ(I gt )||1;

[0173] Therefore, the total loss function for training the FDformer is defined as:

[0174] L stage2 = L pix +λ2L fft +λ3L per ;

[0175] where λ2 is set to 0.1, and λ3 is set to 0.01. The parameters of the Fourier-based decoupling module are adjusted according to the calculated second loss value, so that the trained decoupling network is obtained. In addition, the multi-scale amplitude restoration module needs to input a scaling factor to introduce the expected brightness information. In addition to allowing users to customize the global brightness, the embodiments of the present application also provide a simple method to obtain the reference global brightness. The embodiments of the present application combine the SEBlock and the fully connected layer to infer called the brightness prediction network (LPNet), and the expression of the brightness prediction network is as follows:

[0176]

[0177] where I is the input dark-light blurred image, represents the expected represents the sigmoid function, FC represents the fully connected layer, and Avg 1×1denotes global average pooling, in order to better extract the features of images of different scales while maintaining computational efficiency, the SE adopted by the embodiments of the present application represents a network containing three different scale SEBlocks in series, wherein the SEBlock is a module used in a convolutional neural network (CNN) and is mainly used to implement a Squeeze-and-Excitation (SE) attention mechanism. The real-time exchange rate of the present application adopts an MSE loss to train the brightness prediction network, and the loss function L of the brightness prediction network b The expression of the loss function L is as follows:

[0178]

[0179] In the formula, denotes the ratio of the global brightness of the original input image to the global brightness of the output reference image.

[0180] Next, in combination with a specific application scenario, the scheme of the embodiments of the present application is described and explained in detail:

[0181] The embodiments of the present application are applied to the field of image processing, and specifically can be applied to the restoration of dark light blur images taken by hand in a dark light scene, and can also be applied to camera perception enhancement in a night automatic driving scene. The embodiments of the present application can model the physical degradation process of dark light degradation and blur degradation based on amplitude modulation and phase correlation, respectively, so as to effectively restore dark light blur images. Among them, according to the principle that the global brightness information is contained in the amplitude, a method for customizing the brightness is designed. The embodiments of the present application can also propose an effective frequency domain self-attention mechanism and a feedforward layer according to the principle that dark light degradation is mainly contained in the amplitude and blur degradation is mainly contained in the phase. By designing a special feature extraction method for different degradation features, different degradation information can be effectively obtained, and effective restoration of mixed degradation images can be realized. Through experimental comparison, the PSNR index calculated by the method of the embodiments of the present application on the paired image test set of dark light blur is about 0.4dB higher than that of the current advanced method (top1 PSNR is about 27.79dB), reaching about 28.19dB, and the edges of the blur images can be significantly restored, which is sufficient to prove the effectiveness of the method of the embodiments of the present application.

[0182] The embodiments of the present application also provide an image restoration system based on frequency domain decoupling, which can implement the above-mentioned image restoration method based on frequency domain decoupling. The system comprises:

[0183] A first module is configured to acquire a to-be-processed image and input the to-be-processed image into a decoupling network, wherein the decoupling network comprises a multi-scale amplitude restoration module and a Fourier-based decoupling module.

[0184] The second module is configured to perform multi-scale transformation processing on the image to be processed to obtain a multi-scale image set.

[0185] The third module is configured to input the multi-scale image set into the multi-scale amplitude restoration module to perform amplitude restoration processing to obtain a restored and brightened image set.

[0186] The fourth module is configured to perform component extraction processing on the restored and brightened image set and the image to be processed to obtain prior information.

[0187] The fifth module is configured to input the image to be processed and the prior information into the Fourier-based decoupling module to perform image restoration processing to obtain a restored image.

[0188] It can be understood that the content in the above method embodiments is applicable to the present device embodiments, the present device embodiments specifically implement the same functions as the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0189] The present application also provides a computer device, which comprises a memory and a processor. The memory stores a computer program, and the processor implements the above image restoration method based on frequency domain decoupling when executing the computer program. The computer device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0190] It can be understood that the content in the above method embodiments is applicable to the present device embodiments, the present device embodiments specifically implement the same functions as the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0191] Please refer to Figure 8 , ​ The hardware structure of the computer device of another embodiment is illustrated, which comprises:

[0192] The processor 801 can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is used to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0193] The memory 802 can be implemented in the form of a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM), etc. The memory 802 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the related program codes are stored in the memory 802 and are called and executed by the processor 801 to implement the image restoration method based on frequency domain decoupling provided by the embodiments of the present application;

[0194] The input / output interface 803 is configured to realize information input and output.

[0195] The communication interface 804 is configured to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0196] The bus 805 is configured to transmit information between various components (for example, the processor 801, the memory 802, the input / output interface 803 and the communication interface 804) of the device.

[0197] The processor 801, the memory 802, the input / output interface 803 and the communication interface 804 are connected to each other through the bus 805 to realize the communication connection between the device.

[0198] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the above-mentioned image restoration method based on frequency domain decoupling.

[0199] It can be understood that the contents in the above method embodiments are applicable to the present storage medium embodiments. The functions implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved by the present storage medium embodiments are the same as those of the above method embodiments.

[0200] The memory is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor. These remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0201] The embodiment of the present application provides a kind of based on frequency domain decoupling image restoration method, computer device and storage medium, the scheme is input to multi-scale amplitude restoration module by multi-scale image set, carries out amplitude restoration processing, obtains restoration brightening image set, can be restored to the brightness of image by adjusting to the amplitude;In addition, the scheme is input to Fourier-based decoupling module by the image to be processed and prior information, carries out image restoration processing, obtains restoration image, can be decoupled based on Fourier-based decoupling module for different degradation characteristics, improve the restoration effect of image.

[0202] The embodiment described in the present application is to more clearly illustrate the technical scheme of the embodiment of the present application, and does not constitute a limitation on the technical scheme provided by the embodiment of the present application.

[0203] Those skilled in the art can understand that the technical scheme shown in the figure does not constitute a limitation on the embodiment of the present application, and can include more or fewer steps than the figure, or combine certain steps, or different steps.

[0204] The device embodiment described above is only schematic, wherein the units illustrated as separate components can be or can not be physically separated, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0205] Those skilled in the art can understand that all or some steps in the method disclosed above, the function modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0206] The terms "first", "second", "third", "fourth" and the like (if any) in the specification of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0207] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B, and A and B existing at the same time, wherein A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, and c can be single or multiple.

[0208] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0209] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. According to actual needs, some or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0210] In addition, the functional units in each embodiment of the application can be integrated in one processing unit, or each unit can be physically present, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0211] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0212] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. An image restoration method based on frequency domain decoupling, characterized in that: The method comprises the following steps: Acquire an image to be processed, and input the image to be processed into a decoupling network, wherein the decoupling network includes a multi-scale amplitude restoration module and a Fourier-based decoupling module; Performing multi-scale transformation on the image to be processed to obtain a multi-scale image set; Inputting the multi-scale image set into the multi-scale amplitude restoration module for amplitude restoration processing to obtain a restored and brightened image set; performing component extraction processing on the restored and brightened image set and the image to be processed to obtain prior information; Inputting the image to be processed and the prior information into the Fourier-based decoupling module for image restoration processing to obtain a restored image; The Fourier-based decoupling module includes a decoupled self-attention layer, a decoupled feedforward layer, and a cross-attention feedforward network. The image to be processed and the prior information are input into the Fourier-based decoupling module for image restoration processing to obtain a restored image, including the following steps: Performing feature self-attention calculation processing on the image to be processed based on the decoupled self-attention layer to obtain a first feature vector; Performing frequency learning and spatial learning processing on the first eigenvector based on the decoupled feedforward layer to obtain a second eigenvector; Performing cross-attention calculation processing on the second eigenvector and the prior information based on the cross-attention feedforward network to obtain a third eigenvector; Performing convolution processing on the third eigenvector to obtain a feature map, and performing addition processing on the feature map and the image to be processed to obtain the restored image; Before inputting the image to be processed into the decoupling network, the method further includes training the decoupling network, including the following steps: Get training images; Inputting the training image into the multi-scale amplitude restoration module to perform coarse alignment loss and amplitude loss calculation processing to obtain a first loss value; Adjusting parameters of the multi-scale amplitude restoration module according to the first loss value, and freezing the adjusted parameters; Inputting the training image into the Fourier-based decoupling module to perform pixel loss, frequency loss, and perceptual loss calculations to obtain a second loss value; The parameters of the Fourier-based decoupling module are adjusted according to the second loss value to obtain a trained decoupling network.

2. The method according to claim 1, characterized in that Inputting the multi-scale image set into the multi-scale amplitude restoration module for amplitude restoration processing to obtain a restored and brightened image set comprises the following steps: Get the scaling factor; Gamma correction is performed on the multi-scale image set according to the scaling factor to obtain the restored and brightened image set.

3. The method according to claim 1, characterized in that The step of performing component extraction processing on the restored and brightened image set and the image to be processed to obtain prior information comprises the following steps: performing amplitude component extraction processing on each image in the restored and brightened image set to obtain an amplitude component set; Performing phase component extraction processing on the image to be processed to obtain a phase component; The set of amplitude components and the phase component are determined as the prior information.

4. The method according to claim 1, wherein The step of performing feature self-attention calculation processing on the image to be processed based on the decoupled self-attention layer to obtain a first feature vector includes the following steps: Performing linear transformation processing on the high-dimensional features of the image to be processed to obtain a query vector, a key vector, a value vector and an information vector, wherein the information vector includes amplitude information and phase information; Performing phase correlation calculation processing on the query vector and the key vector according to the information vector to obtain phase correlation attention; Performing amplitude self-attention calculation processing on the query vector and the key vector according to the information vector to obtain amplitude modulated attention; Performing hybrid attention calculation processing on the query vector and the key vector to obtain hybrid attention; The value vector is weightedly calculated according to the phase-related attention, the amplitude-modulated attention and the mixed attention to obtain the first feature vector.

5. The method according to claim 1, characterized in that The decoupled feedforward layer includes a frequency branch and a space branch, and performing frequency learning and space learning processing on the first eigenvector based on the decoupled feedforward layer to obtain a second eigenvector includes the following steps: performing frequency selection and shift processing on the amplitude and phase of the first eigenvector based on the frequency branch to obtain a frequency feature; Performing local feature learning processing in the spatial domain on the first feature vector based on the spatial branch to obtain a spatial feature; Aggregating and nonlinearly transforming the frequency features and the spatial features to obtain the second feature vector.

6. The method according to claim 1, characterized in that The step of performing cross-attention calculation processing on the second eigenvector and the prior information based on the cross-attention feedforward network to obtain the third eigenvector comprises the following steps: Performing a Fourier cross-attention calculation on the second eigenvector and the prior information to obtain an intermediate eigenvector; Perform spatial modulation processing on the intermediate eigenvector to obtain the third eigenvector.

7. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 6 when executing the computer program.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images