Image restoration method based on frequency domain decoupling, computer device and storage medium

Abstract

The invention discloses an image restoration method based on frequency domain decoupling, a computer device and a storage medium, and the method comprises the steps: obtaining a to-be-processed image, and inputting the to-be-processed image to a decoupling network; performing multi-scale transformation processing on the to-be-processed image to obtain a multi-scale image set; inputting the multi-scale image set into a multi-scale amplitude restoration module for amplitude restoration processing to obtain a restored and brightened image set; performing component extraction processing on the restored and brightened image set and the to-be-processed image to obtain prior information; and inputting the to-be-processed image and the prior information to a Fourier-based decoupling module for image restoration processing to obtain a restored image. According to the embodiment of the invention, decoupling processing can be carried out for different degradation features based on the Fourier decoupling module, the image restoration effect is improved, and the method can be widely applied to the technical field of image processing.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image restoration method, a computer device and a storage medium based on frequency domain decoupling. Background Art

[0002] In the related art, there are dark light enhancement and deblurring image restoration methods, that is, image restoration is performed on a specific and single degradation, or an intuitive cascade method is used for image restoration. However, in practical applications, it is found that these methods cannot effectively decouple dark light degradation and blur degradation, and cannot effectively model different degradations, resulting in the inability to restore clear edges in dark light blurred images, affecting the effect of image restoration.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose an image restoration method, a computer device and a storage medium based on frequency domain decoupling, which can improve the ability to restore the clear edges of an image.

[0005] To achieve the above objective, an embodiment of the present application provides an image restoration method based on frequency domain decoupling, the method comprising:

[0006] Acquire an image to be processed, and input the image to be processed into a decoupling network, wherein the decoupling network includes a multi-scale amplitude restoration module and a Fourier-based decoupling module;

[0007] Performing multi-scale transformation processing on the image to be processed to obtain a multi-scale image set;

[0008] Inputting the multi-scale image set into the multi-scale amplitude restoration module for amplitude restoration processing to obtain a restored and brightened image set;

[0009] Performing component extraction processing on the restored and brightened image set and the image to be processed to obtain prior information;

[0010] The image to be processed and the prior information are input into the Fourier-based decoupling module for image restoration processing to obtain a restored image.

[0011] In some embodiments, the step of inputting the multi-scale image set into the multi-scale amplitude restoration module for amplitude restoration processing to obtain a restored and brightened image set comprises the following steps:

[0012] Get the scaling factor;

[0013] Gamma correction is performed on the multi-scale image set according to the scaling factor to obtain the restored and brightened image set.

[0014] In some embodiments, performing component extraction processing on the restored and brightened image set and the image to be processed to obtain prior information includes the following steps:

[0015] Performing amplitude component extraction processing on each image in the restored and brightened image set to obtain an amplitude component set;

[0016] Performing phase component extraction processing on the image to be processed to obtain a phase component;

[0017] The set of amplitude components and the phase component are determined as the prior information.

[0018] In some embodiments, the Fourier-based decoupling module includes a decoupled self-attention layer, a decoupled feedforward layer, and a cross-attention feedforward network, and the image to be processed and the prior information are input into the Fourier-based decoupling module for image restoration processing to obtain a restored image, comprising the following steps:

[0019] Performing feature self-attention calculation processing on the image to be processed based on the decoupled self-attention layer to obtain a first feature vector;

[0020] Performing frequency learning and space learning processing on the first eigenvector based on the decoupled feedforward layer to obtain a second eigenvector;

[0021] Performing cross-attention calculation processing on the second eigenvector and the prior information based on the cross-attention feedforward network to obtain the third eigenvector;

[0022] The third eigenvector is convolved to obtain a feature map, and the feature map is added to the image to be processed to obtain the restored image.

[0023] In some embodiments, performing feature self-attention calculation processing on the image to be processed based on the decoupled self-attention layer to obtain a first feature vector includes the following steps:

[0024] Performing linear change processing on the image to be processed to obtain a query vector, a key vector, a value vector and an information vector, wherein the information vector includes amplitude information and phase information;

[0025] Performing phase correlation calculation processing on the query vector and the key vector according to the information vector to obtain phase correlation attention;

[0026] Performing amplitude self-attention calculation processing on the query vector and the key vector according to the information vector to obtain amplitude modulated attention;

[0027] Performing mixed attention calculation processing on the query vector and the key vector to obtain mixed attention;

[0028] The value vector is weightedly calculated according to the phase-related attention, the amplitude modulation attention and the mixed attention to obtain the first feature vector.

[0029] In some embodiments, the decoupled feedforward layer includes a frequency branch and a space branch, and performing frequency learning and space learning processing on the first feature vector based on the decoupled feedforward layer to obtain a second feature vector includes the following steps:

[0030] Performing frequency selection and shift processing on the amplitude and phase of the first feature vector based on the frequency branch to obtain a frequency feature;

[0031] Based on the spatial branch, the first feature vector is subjected to local feature learning processing in the spatial domain to obtain a spatial feature;

[0032] The frequency feature and the spatial feature are aggregated and nonlinearly transformed to obtain the second feature vector.

[0033] In some embodiments, performing cross-attention calculation processing on the second feature vector and the prior information based on the cross-attention feedforward network to obtain the third feature vector includes the following steps:

[0034] Performing Fourier cross-attention calculation processing on the second feature vector and the prior information to obtain an intermediate feature vector;

[0035] The intermediate eigenvector is subjected to spatial modulation processing to obtain the third eigenvector.

[0036] In some embodiments, before inputting the image to be processed into the decoupling network, the method further includes training the decoupling network, including the following steps:

[0037] Get training images;

[0038] Inputting the training image into the multi-scale amplitude restoration module to perform coarse alignment loss and amplitude loss calculation processing to obtain a first loss value;

[0039] Adjusting parameters of the multi-scale amplitude restoration module according to the first loss value, and freezing the adjusted parameters;

[0040] Inputting the training image into the Fourier-based decoupling module to perform pixel loss, frequency loss and perceptual loss calculation processing to obtain a second loss value;

[0041] The parameters of the Fourier-based decoupling module are adjusted according to the second loss value to obtain a trained decoupling network.

[0042] To achieve the above object, another aspect of the embodiment of the present application provides an image restoration system based on frequency domain decoupling, the system comprising:

[0043] The first module is used to obtain an image to be processed and input the image to be processed into a decoupling network, wherein the decoupling network includes a multi-scale amplitude restoration module and a Fourier-based decoupling module;

[0044] The second module is used to perform multi-scale transformation processing on the image to be processed to obtain a multi-scale image set;

[0045] A third module is used to input the multi-scale image set into the multi-scale amplitude restoration module for amplitude restoration processing to obtain a restored and brightened image set;

[0046] The fourth module is used to perform component extraction processing on the restored and brightened image set and the image to be processed to obtain prior information;

[0047] The fifth module is used to input the image to be processed and the prior information into the Fourier-based decoupling module for image restoration processing to obtain a restored image.

[0048] To achieve the above objective, another aspect of an embodiment of the present application provides a computer device, the computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0049] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0050] The embodiments of the present application include at least the following beneficial effects: the embodiments of the present application provide an image restoration method, a computer device and a storage medium based on frequency domain decoupling. The scheme obtains a restored and brightened image set by inputting a multi-scale image set into a multi-scale amplitude restoration module for amplitude restoration processing, and can adjust the brightness of the image by restoring the amplitude; in addition, the scheme obtains a restored image by inputting the image to be processed and prior information into a Fourier-based decoupling module for image restoration processing. The scheme can perform decoupling processing based on the Fourier-based decoupling module for different degradation features, thereby improving the image restoration effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1is a flow chart of an image restoration method based on frequency domain decoupling provided in an embodiment of the present application;

[0052] Figure 2 (a) is a clear image under normal lighting. Figure 2 (b) is the amplitude diagram corresponding to the clear image under normal lighting. Figure 2 (c) is the phase image corresponding to the clear image under normal lighting. Figure 2 (d) is the normal light blurred image after phase swapping, Figure 2 (e) Yes Figure 2 (d) The local enlarged image corresponding to the red box, Figure 2 (f) is correct Figure 2 (a) Image after histogram equalization processing. Figure 2 (g) Yes Figure 2 (b) The spatial domain representation, Figure 2 (h) is a dark light blurred image, Figure 2 (i) is the amplitude map corresponding to the dark light blurred image, Figure 2 (j) is the phase map corresponding to the dark light blurred image. Figure 2 (k) is the dark light clear image after phase exchange, Figure 2 (l) Yes Figure 2 (k) The red box corresponds to the local enlarged image. Figure 2 (m) is correct Figure 2 (h) The image is obtained after histogram equalization processing. Figure 2 (n) Yes Figure 2 (b) Spatial domain representation;

[0053] Figure 3 It is a schematic diagram of the overall structure of a decoupling network provided in an embodiment of the present application;

[0054] Figure 4 is a structural schematic diagram of a multi-scale amplitude restoration module provided in an embodiment of the present application;

[0055] Figure 5 It is a structural schematic diagram of a decoupled self-attention layer provided in an embodiment of the present application;

[0056] Figure 6 is a structural schematic diagram of a decoupled feedforward layer provided in an embodiment of the present application;

[0057] Figure 7 It is a schematic diagram of the structure of a cross-attention feedforward network provided in an embodiment of the present application;

[0058] Figure 8 It is a schematic diagram of the hardware structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0060] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0061] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0063] In the related art, there are dark light enhancement and deblurring image restoration methods, that is, image restoration is performed on a specific and single degradation, or an intuitive cascade method is used for image restoration. However, in actual applications, it is found that these methods cannot effectively decouple dark light degradation and blur degradation, and cannot effectively model different degradations, resulting in the inability to restore clear edges in dark light blurred images, affecting the image restoration effect. There are also methods in the related art that can handle two types of degradation at the same time, but since these methods fail to effectively model different degradations, the effect is poor, which also affects the image restoration effect.

[0064] For example, the image restoration method can adopt a dark image enhancement method. Common dark image enhancement methods based on traditional image processing technology include methods based on histogram equalization, methods based on retinex theory, and methods based on gamma correction. However, these traditional dark light enhancement algorithms rely heavily on manual prior knowledge and show poor generalization ability and significant color cast when facing diverse scenes. In addition, these methods are often sensitive to noise and easily amplify the noise in the image, thereby affecting the dark light enhancement effect. The dark light enhancement method based on deep learning has a better enhancement effect and can be roughly divided into time domain-based methods and frequency domain-based methods. The time domain-based method directly processes the pixel value of the image, that is, enhances the image in the spatial domain. The existing spatial domain-based methods often ignore the Fourier frequency information, which can capture global properties and have the ability to decouple image degradation and content components. These methods mainly focus on the effect of amplitude on brightness. However, they often ignore the effect of phase, especially in scenes with blur. Therefore, this type of method often cannot restore the blur degradation in the image well in the case of both dark light and blur degradation.

[0065] Exemplarily, the image restoration method can adopt a deblurring method. Traditional image deblurring methods mainly include blind deblurring and non-blind deblurring methods. Blind deblurring methods aim to estimate the blur kernel and the deblurred image simultaneously. However, due to the high ill-posedness of the blind deblurring problem, these methods are usually sensitive to initial conditions and easily fall into local optimality, resulting in unsatisfactory deblurring effects. Non-blind deblurring methods assume that the blur kernel is known and restore the original image through inverse convolution technology. However, in practical applications, it is usually difficult to accurately obtain the blur kernel, so the actual effects of these methods are often seriously affected by the blur kernel estimation error. Deblurring methods based on deep learning have more powerful learning capabilities and can adaptively learn blur degradation information. However, these methods ignore the dark light degradation that may exist simultaneously in actual scenes. In a dark light environment, problems such as pixel saturation noise in high brightness areas are very unfavorable for deblurring.

[0066] Exemplarily, the image restoration method can also adopt a joint dark light enhancement and deblurring method. Most of the current research on image restoration focuses on specific and single degradations. However, in actual scenes, images are often affected by multiple degradations. Simply adopting an intuitive cascade method may lead to poor results, resulting in error accumulation and increased computational burden. The basic assumption of the joint dark light enhancement and deblurring method is that the downsampled blurred image can be approximated as a downsampled clear image. However, due to the interdependence of dark light and blur degradation in the spatial domain, different degradations are difficult to effectively decouple, which limits the performance of the designed module and has limitations in restoring sharp edges. Therefore, in the related art, the methods of single dark light enhancement, single deblurring, and joint dark light enhancement and deblurring cannot effectively decouple dark light degradation and blur degradation, thereby failing to restore clear edges in dark light blurred images.

[0067] In view of this, an image restoration method, a computer device and a storage medium based on frequency domain decoupling are provided in the embodiments of the present application. The scheme decouples dark light degradation and blur degradation in the Fourier domain, and can model the physical degradation processes of dark light degradation and blur degradation based on amplitude modulation and phase correlation respectively. By using a Fourier-based decoupling network for joint dark light enhancement and deblurring, dark light degradation and blur degradation can be effectively decoupled and modeled, thereby significantly improving the ability to restore the clear edges of the image.

[0068] An image restoration method based on frequency domain decoupling provided in an embodiment of the present application relates to the field of image processing technology. An image restoration method based on frequency domain decoupling provided in an embodiment of the present application can be applied to a terminal, can be applied to a server, or can be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements an image restoration method based on frequency domain decoupling, etc., but is not limited to the above forms.

[0069] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer computer devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0070] Figure 1 is an optional flow chart of an image restoration method based on frequency domain decoupling provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S105.

[0071] Step S101, obtaining an image to be processed, and inputting the image to be processed into a decoupling network, wherein the decoupling network includes a multi-scale amplitude restoration module and a Fourier-based decoupling module;

[0072] Step S102, performing multi-scale transformation processing on the image to be processed to obtain a multi-scale image set;

[0073] Step S103, inputting the multi-scale image set into the multi-scale amplitude restoration module for amplitude restoration processing to obtain a restored and brightened image set;

[0074] Step S104, performing component extraction processing on the restored and brightened image set and the image to be processed to obtain prior information;

[0075] Step S105: input the image to be processed and the prior information into the Fourier-based decoupling module for image restoration processing to obtain a restored image.

[0076] In the steps S101 to S105 shown in the embodiment of the present application, the processed image is restored by a Fourier-based decoupling network, and dark light enhancement and deblurring can be performed on the processed image. In the embodiment of the present application, it is found that dark light degradation and blur degradation can be manifested in the amplitude and phase components of the image respectively. The two degradations are further analyzed and it is found that dark light degradation has different degradation forms in different frequency bands of the amplitude, while blur degradation can be modeled by phase correlation. Please refer to Figure 2(a) and Figure 2 (h) are the clear image under normal lighting and the blurred image under dark lighting, Figure 2 (b) Figure 2 (c) Figure 2 (i) and Figure 2 (j) are their corresponding amplitude diagram and phase diagram. The embodiment of the present application obtains a dark light clear image by exchanging their phases. Figure 2 (k) and a normal light blurred image Figure 2 (d). Figure 2 (f) and Figure 2 (m) are respectively processed by histogram equalization Figure 2 (c) and Figure 2 (j) The spatial domain representation further proves that the phase can represent the complete structural information. In addition, Figure 2 (g) and Figure 2 (n) are Figure 2 (b) and Figure 2 (i) The spatial domain representation. It can be observed that Figure 2 (g) Ratio Figure 2 (n) are brighter, but they do not reveal structural information. This shows that the phase mainly encodes structural information, while the amplitude mainly encodes brightness information. Further, by exchanging specific frequency bands of amplitude and phase in the image, the embodiment of the present application further explores the characteristics of different frequency bands in amplitude and phase. For two images A and B, when the proportion of the phase of image B in the exchanged image is high, the structure of B is obviously retained in the image, with the detail information of B, while the information of A is reflected as a blurred outline. As the proportion of phase B decreases, the structure of A gradually becomes clear and complete, while the contribution of B is reduced to edge information. This shows that the high-frequency phase mainly encodes edge information, while the low-frequency phase mainly encodes the overall structure of the image. When the proportion of the amplitude of image B in the exchanged image is low, the image shows an overall brightness similar to that of image B, but with lower contrast. As the amplitude ratio of image B in the exchanged image increases, the color and contrast of the detail area gradually improve, approaching the effect of image B. This shows that the low-frequency amplitude mainly encodes the overall brightness and contrast of the image, while the high-frequency amplitude encodes the color information of the detail area.

[0077] Therefore, the embodiment of the present application constructs a decoupling network based on the transformers architecture, and can restore the image through the decoupling network in combination with dark light enhancement and deblurring. Among them, the decoupling network includes a multi-scale amplitude restoration module, and the amplitude restoration is performed through the multi-scale amplitude restoration module. It can take into account the needs of personalized restoration, customize the brightness by adjusting the coarse amplitude of the image, and use the restoration result as the prior of the Fourier-based decoupling module through the Fourier-based cross-attention mechanism. Among them, the Fourier-based decoupling module can learn different representations of different degradations through the Fourier domain self-attention mechanism. In addition, considering that different frequency ranges contain unique information in terms of amplitude and phase, a filtering mechanism is introduced to amplify or attenuate specific frequency bands of features. The embodiment of the present application can effectively decouple and model dark light degradation and blur degradation, thereby significantly improving the ability to restore clear edges of images.

[0078] In step S101 of some embodiments, the image to be processed can be obtained by uploading by the user, or by other methods, such as obtaining the image to be processed from a database, but is not limited thereto. The image to be processed is an image that needs to be subjected to dark light enhancement and deblurring processing, and then the image to be processed is input into a Fourier-based decoupling network. Figure 3 The decoupling network consists of a multi-scale amplitude restoration module (MAR) and a Fourier-based decoupling module (FDformer), in which the Fourier-based decoupling module consists of a Fourier-based decoupled self-attention layer (FDSA), a Fourier-based decoupled feedforward layer (FDFFN), and a Fourier-based cross-attention feedforward network (FCAFFN).

[0079] In step S102 of some embodiments, multi-scale transformation processing is performed on the image to be processed to obtain a multi-scale image set.

[0080] In the embodiment of the present application, by performing multiple scale transformations on the image to be processed, images at different scales are obtained. For example, the image to be processed is scaled proportionally to obtain multiple images of different scales, and the image to be processed and images of other scales are used as a multi-scale image set. These images are used to input the multi-scale amplitude restoration module for amplitude restoration processing, and the restored results are used as prior knowledge to guide the Fourier-based decoupling module to perform brightness restoration, thereby improving the effect of image restoration.

[0081] In step S103 of some embodiments, the multi-scale image set is input into the multi-scale amplitude restoration module for amplitude restoration processing to obtain a restored and brightened image set.

[0082] In an embodiment of the present application, the multi-scale image set processed by the multi-scale transformation in the above steps is input into a multi-scale amplitude restoration module for amplitude restoration processing. The amplitude can be roughly restored by the multi-scale amplitude restoration module, so that the restored rough amplitude can be used as a brightness prior, thereby guiding the Fourier-based decoupling module to focus on restoring details and improving the image restoration effect.

[0083] In step S103 of some embodiments, the multi-scale image set is input into the multi-scale amplitude restoration module for amplitude restoration processing to obtain a restored and brightened image set, which includes the following steps:

[0084] Get the scaling factor;

[0085] Gamma correction is performed on the multi-scale image set according to the scaling factor to obtain the restored and brightened image set.

[0086] In the embodiment of the present application, the scaling factor represents the ratio of the global brightness of the original input image to the global brightness expected by the user, and is used to adjust the intermediate features of the image. The amplitude component of the image can be adjusted according to the scaling factor to adjust the image brightness, thereby realizing user-defined image brightness. Figure 4 The multi-scale amplitude restoration module adopts the U-Net architecture, uses FP blocks and convolutional layers to extract features, and processes each image in the multi-scale image set. For each scale, the embodiment of the present application sets a pixel-level correction factor γ n , whose value range is (0,1). Gamma correction is applied to the input image to restore the rough brightness and obtain the restored brightened image. The expression of the restored brightened image is as follows:

[0087]

[0088] In the formula, I n represents the image of the nth scale, γ n represents the pixel-level correction factor, represents the restored brightened image of the corresponding nth scale. The brightness information can be obtained by extracting the amplitude component of the restored brightened image. In addition, for a dark light blurred image, the brightness of the corresponding restored result is not unique. Therefore, the embodiment of the present application encodes the custom brightness in the rough amplitude. Specifically, for a grayscale image f(x,y) with a scale of H×W, its amplitude component |F(0,0)| is expressed as That is, the average value of all pixel values ​​in the image f(x,y). In the embodiment of the present application, the amplitude component |F(0,0)| is regarded as the global brightness of the image f(x,y). Similarly, for an RGB image, the global brightness can be calculated by converting it into a grayscale image.

[0089] In order to study the influence of the amplitude component |F(0,0)| on the entire image f(x,y), |F(0,0)| is amplified by a factor α to generate a new image f′(x,y):

[0090]

[0091] Wherein, H represents the height of the image, W represents the width of the image, and F(u,v) represents the Fourier transform of the image f(x,y). This operation is equivalent to increasing the value of each pixel of the image in the spatial domain without considering the color distribution, resulting in color distortion. Therefore, the embodiment of the present application considers using an implicit method to customize the brightness. In the intermediate features of each scale image, a scaling factor is introduced to adjust the intermediate features by representing the ratio of the global brightness of the original input image to the global brightness expected by the user and through gamma correction, so that the user-defined brightness can be achieved in the amplitude component of the restored brightened image.

[0092] In step S104 of some embodiments, performing component extraction processing on the restored and brightened image set and the image to be processed to obtain prior information includes the following steps:

[0093] Performing amplitude component extraction processing on each image in the restored and brightened image set to obtain an amplitude component set;

[0094] Performing phase component extraction processing on the image to be processed to obtain a phase component;

[0095] The set of amplitude components and the phase component are determined as the prior information.

[0096] In an embodiment of the present application, the amplitude component of the restored brightened image and the phase component of the original dark light blurred image are extracted respectively, and the amplitude component set and the phase component are determined as prior information, so that the extracted amplitude component and phase component can be input as prior information into a Fourier-based cross-attention feedforward network to guide the decoupling module to perform image restoration, thereby improving the processing efficiency of image restoration.

[0097] In some implementations of step S105, the step of inputting the image to be processed and the prior information into the Fourier-based decoupling module for image restoration processing to obtain a restored image includes the following steps:

[0098] Performing feature self-attention calculation processing on the image to be processed based on the decoupled self-attention layer to obtain a first feature vector;

[0099] Based on the decoupled feedforward layer, frequency selection and offset learning processing are performed on the first eigenvector in a high-dimensional space to obtain a second eigenvector;

[0100] Performing cross-attention calculation processing on the second feature vector and the prior information based on the cross-attention feedforward network to obtain a third feature vector;

[0101] The third eigenvector is convolved to obtain a feature map, and the feature map is added to the image to be processed to obtain the restored image.

[0102] In an embodiment of the present application, a Fourier-based decoupling module includes a decoupled self-attention layer, a decoupled feedforward layer, and a cross-attention feedforward network. Based on the decoupled self-attention layer, the processed image is subjected to feature self-attention calculation processing, and different representations of blur degradation and dark degradation can be obtained through self-attention to obtain a first eigenvector. Since dark light degradation and blur degradation have different representations in different frequency bands, the embodiment of the present application processes the first eigenvector through a Fourier decoupled feedforward layer, and can adaptively learn frequency selection and offset in a high-dimensional space. In addition, the introduction of multi-scale features of the image can improve the effect of image deblurring, and the multi-scale phase features and the restored rough amplitude features are cross-attention calculated by a Fourier cross-attention feedforward layer (FCAFFN) with a Fourier cross-attention (FCA) mechanism, and the restored image is finally output, which can effectively decouple and model dark light degradation and blur degradation, thereby significantly improving the ability to restore the clear edges of the image.

[0103] In some embodiments, performing feature self-attention calculation processing on the image to be processed based on the decoupled self-attention layer to obtain a first feature vector includes the following steps:

[0104] Performing linear change processing on the high-dimensional features of the image to be processed to obtain a query vector, a key vector, a value vector and an information vector, wherein the information vector includes amplitude information and phase information;

[0105] Performing phase correlation calculation processing on the query vector and the key vector according to the information vector to obtain phase correlation attention;

[0106] Performing amplitude self-attention calculation processing on the query vector and the key vector according to the information vector to obtain amplitude modulated attention;

[0107] Performing mixed attention calculation processing on the query vector and the key vector to obtain mixed attention;

[0108] The value vector is weightedly calculated according to the phase-related attention, the amplitude modulation attention and the mixed attention to obtain the first feature vector.

[0109] In the embodiment of the present application, the decoupled self-attention layer is used to obtain different representations of blur degradation and dark degradation through self-attention. Figure 5 After the decoupled self-attention layer normalizes the processed image, it is linearly transformed through 1*1 convolution and 3*3 depth-separable convolution to obtain the query vector, key vector, value vector and information vector, where the information vector provides additional amplitude and phase information for phase-correlated attention and amplitude modulation attention. According to the information vector, the query vector and the key vector are phase-correlated and processed to obtain the phase-correlated attention. The following is a derivation and analysis of the process of calculating the phase-correlated attention. Specifically, for blur degradation, blur can be approximated by averaging a series of consecutive frames, and the expression is as follows:

[0110]

[0111] in, represents the observed blurred image, g represents the mapping of the potential clear signal to the observed camera response function, m represents the total number of frames during image capture, and S[i] represents the clear signal of the i-th frame. The embodiment of the present application obtains blur information through a pure phase method. The embodiment of the present application assumes First, considering the case of global motion, the above formula can be converted to:

[0112]

[0113] Among them, △x i represents the displacement relative to the first frame, so that Represents the image of the i-th frame.

[0114] Since the exposure time is usually not too long, the difference between adjacent frames △x i+1 -△x i The embodiment of the present application may assume that this process is approximately a uniform linear motion, and the displacement △x between adjacent frames is consistent, that is, △x i+1 -△x i =△x. Therefore, the above formula can be converted to:

[0115]

[0116] In combination with the Fourier shift theorem, the embodiment of the present application performs Fourier transform processing on the above formula:

[0117]

[0118] Among them, A i (ω) and P i(ω) represent the amplitude and phase respectively. △P(ω)=2π△xω represents the phase difference between adjacent frames, and P1(ω) is the phase of the first frame. Here, the embodiment of the present application only considers the representation of blur in phase, so A i (ω) is set to 1. Using the geometric series summation formula, we can get:

[0119]

[0120] Here, the embodiment of the present application assumes the phases of two images blur1 and blur2 with the same initial phase but different parameters m. The phase correlation between blur1 and blur2 can be expressed as:

[0121]

[0122] Where m1 and m2 represent the blur levels of blur1 and blur2 respectively, and there is a small difference △m between the two (m1=m2+△m,△m→0). Therefore, the embodiment of the present application can define the function G(m)=ln(e jm△P(ω) -1), since △m→0, the formula can be converted to:

[0123]

[0124] Considering that G(m) is a monotonically increasing function of m, and mΔP(ω) represents the phase difference between the first frame and the last frame, G(m) can be regarded as a function related to the blur degree. Therefore, the phase correlation between two blurred images can reflect their relative blur degrees. The embodiment of the present application integrates phase correlation into the self-attention mechanism to explore the representation of blur information. In the Fourier domain, the query vector and key vector The dot product of can be used as self-attention calculation to get self-attention Self-Attention The calculation formula is as follows:

[0125]

[0126] Among them, amp and phase represent the amplitude and phase components of the variable respectively. and The phase relationship between them does not change the mathematical characteristics of the self-attention calculation. In the embodiment of the present application, and Set to a constant of 1 and Thus, the embodiment of the present application can obtain a new form of self-attention, and obtain the first attention The formula is as follows:

[0127]

[0128] According to the Fourier shift theorem, setting the phase to a negative value is equivalent to rotating the matrix 180 degrees in the spatial domain, which still maintains the self-attention mechanism. and The weighted sum is calculated. This shows and The phase correlation between has the mathematical properties of self-attention. It can be regarded as an amplitude-free self-attention calculation, which is robust to changes in illumination. This property makes it possible to effectively extract blur-degraded information even in low-light environments.

[0129] In order to extract dark light degradation information, the embodiment of the present application extracts the amplitude-based self-attention mechanism. Considered as This operation is similar to Assume that it is a filter used to amplify or attenuate The frequency range in the second self-attention calculation formula can be obtained as follows:

[0130]

[0131] Similarly, it can be found It also has the mathematical properties of self-attention calculation.

[0132] In the Fourier domain, amplitude represents the intensity of different frequency components in the image, and a larger amplitude means that the corresponding frequency has greater energy. The phase measures the degree of displacement of the wave of each frequency relative to its starting position. In the self-attention calculation, only the phase is involved, which is equivalent to shifting the plane wave of the same intensity but different frequencies. Although the structural information is retained, it loses the details of the wave intensity, making it difficult to identify the key feature positions. Since the self-attention calculation involving only amplitude cannot represent valid image information, it does not contain any image-related structure. For this reason, the embodiment of the present application introduces an additional branch ε in the above self-attention calculation to supplement the missing amplitude and phase information, namely the information vector. Based on the information vector, the calculation formulas of the first attention and the second attention are modified to obtain the phase-related attention. Amplitude Modulation of Attention and mixed attention The calculation formula is as follows:

[0133]

[0134] in, and and This mechanism is explored in and When blurring the information between , it avoids introducing features related to dark light degradation. Similarly, Main exploration and The amplitude characteristics related to dark light degradation between and Structure-related features. At the same time, and The embodiment of the present application performs feature self-attention calculation processing on the image to be processed by decoupling the self-attention layer, and can design a special feature extraction method for different degradation features, effectively obtain different degradation information, and realize effective restoration of mixed degraded images.

[0135] In some embodiments, the decoupled feedforward layer includes a frequency branch and a space branch, and the method of performing frequency selection and offset learning processing on the first feature vector in a high-dimensional space based on the decoupled feedforward layer to obtain a second feature vector includes the following steps:

[0136] Performing frequency selection and shift processing on the amplitude and phase of the first feature vector based on the frequency branch to obtain a frequency feature;

[0137] Based on the spatial branch, the first feature vector is subjected to local feature learning processing between channels to obtain a spatial feature;

[0138] The frequency feature and the spatial feature are aggregated and nonlinearly transformed to obtain the second feature vector.

[0139] In the embodiment of the present application, since dark light and blur degradation have different representations in different frequency bands, the embodiment of the present application can adaptively learn frequency selection and offset in high-dimensional space through the Fourier decoupled feed-forward layer (FDFFN), see Figure 6 The Fourier decoupled feedforward layer includes a frequency branch and a spatial branch, wherein the frequency branch uses a learnable filter to perform frequency selection and offset operations on the amplitude and phase of the feature respectively. Specifically, the embodiment of the present application divides the first feature vector into p×p small blocks through a patch segmentation operation, and then uses a learnable filter in the Fourier domain to adjust the amplitude and phase information to obtain the frequency feature Let x n Represents the input features at scale n, and the expression of frequency features is as follows:

[0140]

[0141] Among them, PM p×p () indicates the patch merging operation, PSp×p (x n ) represents a patch segmentation operation. and are learnable parameters, which are initialized to 2π and 1 respectively, indicating that the phase and amplitude will not be changed, and feature modulation learning is performed from the initial feature space.

[0142] In the spatial domain branch, the embodiment of the present application implements local feature learning between channels through two 3x3 deep convolutions and GELU nonlinear activation functions to obtain spatial features. Finally, after aggregating spatial features and frequency features through additive fusion, a nonlinear transformation is introduced through a gating mechanism, and then a 1x1 convolution is used to restore the dimension to the original size to obtain the second eigenvector. The expression of the second eigenvector is as follows:

[0143]

[0144] In the formula, represents the second eigenvector, represents 1x1 convolution, Gate represents the gating mechanism, Represents spatial features, The embodiment of the present application can adaptively learn frequency selection and offset in high-dimensional space by decoupling the feedforward layer, thereby improving the effect of image restoration.

[0145] In some embodiments, performing cross-attention calculation processing on the second feature vector and the prior information based on the cross-attention feedforward network to obtain the third feature vector includes the following steps:

[0146] Performing Fourier cross-attention calculation processing on the second feature vector and the prior information to obtain an intermediate feature vector;

[0147] The intermediate eigenvector is subjected to spatial modulation processing to obtain the third eigenvector.

[0148] In the embodiments of this application, please refer to Figure 7 , the second eigenvector is normalized and Fourier transformed to obtain the amplitude variable and phase variable, and the Fourier cross attention calculation is performed in combination with the prior information. The formula for the cross attention calculation is as follows:

[0149]

[0150] Normalization of the display layer, 1×1 convolution Used to adjust the phase and amplitude to the same dimensions as the feature. For learning input image I n Relative blur information between the second eigenvector and the brightened image It is used to modulate the amplitude Here, the embodiment of this application adopts Instead of the phase after restoration by the multi-scale amplitude restoration module Because in the absence of blurred images under normal light, the restoration process of the multi-scale amplitude restoration module may cause damage to the phase information.

[0151] In addition, considering the matching problem between amplitude and phase, simply combining the amplitude of a clear image under normal light with the phase of a blurred image under dark light may produce artifacts. To assist in the adaptive alignment of amplitude and phase, the embodiment of the present application introduces an additional spatial modulation mechanism, and the formula for spatial modulation is as follows:

[0152]

[0153] in, Represents the output feature of the nth scale. After normalizing the output feature, a third feature vector can be obtained. The embodiment of the present application can adaptively align the amplitude and phase of the image feature through the cross-attention feedforward network, thereby improving the image restoration effect.

[0154] In some embodiments, before inputting the image to be processed into the decoupling network, the method further includes training the decoupling network, including the following steps:

[0155] Get training images;

[0156] Inputting the training image into the multi-scale amplitude restoration module to perform coarse alignment loss and amplitude loss calculation processing to obtain a first loss value;

[0157] Adjusting parameters of the multi-scale amplitude restoration module according to the first loss value, and freezing the adjusted parameters;

[0158] Inputting the training image into the Fourier-based decoupling module to perform pixel loss, frequency loss and perceptual loss calculation processing to obtain a second loss value;

[0159] The parameters of the Fourier-based decoupling module are adjusted according to the second loss value to obtain a trained decoupling network.

[0160] In the embodiment of the present application, the decoupling network adopts a two-stage training process, first training the multi-scale amplitude restoration module, then freezing its parameters, and then training the Fourier-based decoupling module. Specifically, in the training process of the multi-scale amplitude restoration module, two loss functions are introduced: the coarse alignment loss L and the coarse alignment loss L. coarse and amplitude loss L amp. Among them, the coarse alignment loss is intended to provide a rough amplitude of brightness and contrast. Due to the lack of normal illumination blurred images as intermediate controls during training, the embodiment of the present application introduces a more flexible supervision method instead of pixel-by-pixel supervision. The calculation formula of the coarse alignment loss is as follows:

[0161]

[0162] in, Indicates that the multi-scale amplitude restoration module is Output on a scale. Represents the average pooling operation of downsampling n-1 times. Where n represents the different scales of the multi-scale amplitude restoration module. is with The same scale output reference image. For the input feature map (I n ), output feature map The value at each position (i, j, c) is the average value of the elements in the pooling window, and the pooling window size is 2 n-1 ×2 n-1 , the expression of the average pooling operation is as follows:

[0163]

[0164] The amplitude loss is used to provide a rough amplitude restoration. The embodiment of the present application introduces an additional loss function to directly supervise the amplitude. The expression of the amplitude loss is as follows:

[0165]

[0166] Therefore, the overall loss function for the training phase of the multi-scale amplitude restoration module can be expressed as the first loss L stage1 , the expression of the first loss is as follows:

[0167] L stage1 =L course +λ1L amp ;

[0168] Here, λ1 is set to 0.01 to balance the two losses.

[0169] Then, the parameters of the multi-scale amplitude restoration module are adjusted according to the calculated first loss value, and the adjusted parameters are frozen. Then, the training image is input into the Fourier-based decoupling module to calculate the pixel loss, frequency loss and perceptual loss to obtain the second loss value. The loss function of the Fourier-based decoupling module consists of pixel loss, frequency loss and perceptual loss. The pixel loss L pixEquivalent to L1 loss, frequency loss is used to supervise the spectrum of the output result in the Fourier frequency domain, and frequency loss L fft The expression is as follows:

[0170]

[0171] Among them, Real and Imag represent the real part and imaginary part in Fourier domain respectively. Indicates I out The two-dimensional discrete Fourier transform of Indicates I gt The two-dimensional discrete Fourier transform of .

[0172] The perceptual loss is supervised by the pre-trained VGG19 network Φ to improve the perceptual quality. The perceptual loss L per The expression is as follows:

[0173] L per =||Φ(I out )-Φ(I gt )||1;

[0174] Therefore, the total loss function for training FDformer is defined as:

[0175] L stage2 =L pix +λ2L fft +λ3L per ;

[0176] Among them, λ2 is set to 0.1 and λ3 is set to 0.01. According to the calculated second loss value, the parameters of the Fourier-based decoupling module are adjusted to obtain the trained decoupling network. In addition, the multi-scale amplitude restoration module needs to input the scaling factor To introduce the expected brightness information. In addition to allowing users to customize the global brightness, the present application embodiment also provides a simple method to obtain the reference global brightness. The present application embodiment combines SEBlock and the fully connected layer to infer It is called the brightness prediction network (LPNet). The expression of the brightness prediction network is as follows:

[0177]

[0178] Where I is the input dark light blurred image, Indicates expected represents the sigmoid function, FC represents the fully connected layer, Avg 1×1Represents global average pooling. In order to better extract features of images of different scales while maintaining computational efficiency, the SE used in the embodiment of the present application represents a network containing three SEBlocks of different scales connected in series, wherein SEBlock is a module used in convolutional neural networks (CNNs) and is mainly used to implement the Squeeze-and-Excitation (SE) attention mechanism. The real-time exchange rate of the present application uses MSE loss to train the brightness prediction network. The loss function L of the brightness prediction network is b The expression is as follows:

[0179]

[0180] In the formula, Represents the ratio of the global brightness of the original input image to the global brightness of the output reference image.

[0181] Below, in combination with specific application scenarios, the solution of the embodiment of the present application is described in detail and explained:

[0182] The embodiments of the present application are applied to the field of image processing technology. Specifically, they can be applied to the restoration of dark-light blurry images taken by hand in dark-light scenes, and can also be applied to image restoration application scenarios such as camera perception enhancement in nighttime autonomous driving scenes. The embodiments of the present application can model the physical degradation processes of dark-light degradation and blur degradation based on amplitude modulation and phase correlation, respectively, so as to effectively restore dark-light blurry images. According to the principle that global brightness information is contained in amplitude, a method for customizing brightness is designed. The embodiments of the present application can also propose an effective frequency-domain self-attention mechanism and a feedforward layer based on the principle that dark-light degradation is mainly contained in amplitude and blur degradation is mainly contained in phase. By designing special feature extraction methods for different degradation features, different degradation information can be effectively obtained to achieve effective restoration of mixed degraded images. After experimental comparison, the PSNR index calculated by the method of the embodiment of the present application on the dark light blurred paired image test set is about 0.4dB higher than the current advanced method (top1 PSNR is about 27.79dB), reaching about 28.19dB, and can significantly restore the edges of the blurred image, which is sufficient to prove the effectiveness of the method of the embodiment of the present application.

[0183] The embodiment of the present application further provides an image restoration system based on frequency domain decoupling, which can implement the above-mentioned image restoration method based on frequency domain decoupling, and the system includes:

[0184] The first module is used to obtain an image to be processed and input the image to be processed into a decoupling network, wherein the decoupling network includes a multi-scale amplitude restoration module and a Fourier-based decoupling module;

[0185] The second module is used to perform multi-scale transformation processing on the image to be processed to obtain a multi-scale image set;

[0186] A third module is used to input the multi-scale image set into the multi-scale amplitude restoration module for amplitude restoration processing to obtain a restored and brightened image set;

[0187] The fourth module is used to perform component extraction processing on the restored and brightened image set and the image to be processed to obtain prior information;

[0188] The fifth module is used to input the image to be processed and the prior information into the Fourier-based decoupling module for image restoration processing to obtain a restored image.

[0189] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0190] The embodiment of the present application also provides a computer device, the computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned image restoration method based on frequency domain decoupling when executing the computer program. The computer device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0191] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0192] See also Figure 8 , Figure 8 The hardware structure of a computer device according to another embodiment is shown, and the computer device includes:

[0193] The processor 801 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0194] The memory 802 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 802, and the processor 801 calls and executes an image restoration method based on frequency domain decoupling in the embodiment of the present application;

[0195] Input / output interface 803, used to implement information input and output;

[0196] The communication interface 804 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0197] A bus 805 that transmits information between the various components of the device (e.g., the processor 801, the memory 802, the input / output interface 803, and the communication interface 804);

[0198] The processor 801 , the memory 802 , the input / output interface 803 and the communication interface 804 are connected to each other in communication within the device via a bus 805 .

[0199] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned image restoration method based on frequency domain decoupling is implemented.

[0200] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0201] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0202] The embodiments of the present application provide an image restoration method, a computer device and a storage medium based on frequency domain decoupling. The scheme obtains a restored and brightened image set by inputting a multi-scale image set into a multi-scale amplitude restoration module for amplitude restoration processing, and can adjust the brightness of the image by restoring the amplitude. In addition, the scheme obtains a restored image by inputting the image to be processed and prior information into a Fourier-based decoupling module for image restoration processing. The Fourier-based decoupling module can perform decoupling processing on different degradation features, thereby improving the image restoration effect.

[0203] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0204] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0205] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0206] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0207] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0208] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0209] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0210] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0211] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0212] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0213] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. An image restoration method based on frequency domain decoupling, characterized in that: The method comprises the following steps: Acquire an image to be processed, and input the image to be processed into a decoupling network, wherein the decoupling network includes a multi-scale amplitude restoration module and a Fourier-based decoupling module; Performing multi-scale transformation processing on the image to be processed to obtain a multi-scale image set; Inputting the multi-scale image set into the multi-scale amplitude restoration module for amplitude restoration processing to obtain a restored and brightened image set; Performing component extraction processing on the restored and brightened image set and the image to be processed to obtain prior information; The image to be processed and the prior information are input into the Fourier-based decoupling module for image restoration processing to obtain a restored image.

2. The method according to claim 1, characterized in that The step of inputting the multi-scale image set into the multi-scale amplitude restoration module for amplitude restoration processing to obtain a restored and brightened image set comprises the following steps: Get the scaling factor; Gamma correction is performed on the multi-scale image set according to the scaling factor to obtain the restored and brightened image set.

3. The method according to claim 1, characterized in that The step of performing component extraction processing on the restored and brightened image set and the image to be processed to obtain prior information comprises the following steps: Performing amplitude component extraction processing on each image in the restored and brightened image set to obtain an amplitude component set; Performing phase component extraction processing on the image to be processed to obtain a phase component; The set of amplitude components and the phase component are determined as the prior information.

4. The method according to claim 1, characterized in that: The Fourier-based decoupling module includes a decoupled self-attention layer, a decoupled feedforward layer and a cross-attention feedforward network. The image to be processed and the prior information are input into the Fourier-based decoupling module for image restoration processing to obtain a restored image, including the following steps: Based on the decoupled self-attention layer, feature self-attention calculation processing is performed on the image to be processed to obtain a first feature vector; Performing frequency learning and space learning processing on the first eigenvector based on the decoupled feedforward layer to obtain a second eigenvector; Performing cross-attention calculation processing on the second eigenvector and the prior information based on the cross-attention feedforward network to obtain the third eigenvector; The third eigenvector is convolved to obtain a feature map, and the feature map is added to the image to be processed to obtain the restored image.

5. The method according to claim 4, characterized in that The step of performing feature self-attention calculation processing on the image to be processed based on the decoupled self-attention layer to obtain a first feature vector includes the following steps: Performing linear change processing on the high-dimensional features of the image to be processed to obtain a query vector, a key vector, a value vector and an information vector, wherein the information vector includes amplitude information and phase information; Performing phase correlation calculation processing on the query vector and the key vector according to the information vector to obtain phase correlation attention; Performing amplitude self-attention calculation processing on the query vector and the key vector according to the information vector to obtain amplitude modulated attention; Performing mixed attention calculation processing on the query vector and the key vector to obtain mixed attention; The value vector is weightedly calculated according to the phase-related attention, the amplitude modulation attention and the mixed attention to obtain the first feature vector.

6. The method according to claim 4, characterized in that The decoupled feedforward layer includes a frequency branch and a space branch, and the step of performing frequency learning and space learning processing on the first feature vector based on the decoupled feedforward layer to obtain a second feature vector includes the following steps: Performing frequency selection and shift processing on the amplitude and phase of the first feature vector based on the frequency branch to obtain a frequency feature; Based on the spatial branch, the first feature vector is subjected to local feature learning processing in the spatial domain to obtain a spatial feature; The frequency feature and the spatial feature are aggregated and nonlinearly transformed to obtain the second feature vector.

7. The method according to claim 4, characterized in that The step of performing cross-attention calculation processing on the second feature vector and the prior information based on the cross-attention feedforward network to obtain the third feature vector comprises the following steps: Performing Fourier cross-attention calculation processing on the second eigenvector and the prior information to obtain an intermediate eigenvector; The intermediate eigenvector is subjected to spatial modulation processing to obtain the third eigenvector.

8. The method according to claim 1, characterized in that: Before inputting the image to be processed into the decoupling network, the method further includes training the decoupling network, including the following steps: Get training images; Inputting the training image into the multi-scale amplitude restoration module to perform coarse alignment loss and amplitude loss calculation processing to obtain a first loss value; Adjusting parameters of the multi-scale amplitude restoration module according to the first loss value, and freezing the adjusted parameters; Inputting the training image into the Fourier-based decoupling module to perform pixel loss, frequency loss and perceptual loss calculation processing to obtain a second loss value; The parameters of the Fourier-based decoupling module are adjusted according to the second loss value to obtain a trained decoupling network.

9. A computer device, characterized in that: The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images