Image processing method and device, equipment and storage medium

Abstract

The invention discloses an image processing method and device, equipment and a storage medium. The method comprises the steps that a first image is acquired; the first image is restored based on an image restoration model to obtain a second image, the image restoration model comprises a frequency domain processing module and a spatial domain processing module, the frequency domain processing module is used for processing the first image from a frequency domain, and the spatial domain processing module is used for processing the first image from a spatial domain to obtain a second image; the second image is a restored image of the first image, and the quality of the second image is higher than that of the first image; and outputting the second image. Therefore, by means of the image restoration model, the first image can be processed from the angles of the spatial domain and the frequency domain, the frequency domain features are dynamically learned while the spatial domain features are mined, and the problem of middle image information loss caused by restoration through a single-angle model is solved. Therefore, the first image can be effectively restored to obtain the second image, and the effect of improving the quality of the image is achieved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, apparatus, device and storage medium. Background Art

[0002] In the bank's counter business process, it is often necessary to use equipment outside the counter, such as high-definition cameras or scanners, to collect identification information and other supporting documents provided by customers or credential documents generated during business processing, and convert these documents into images for storage.

[0003] However, in the process of image acquisition by these devices, the imaging system will be affected by various factors, such as sensor noise, relative movement between the object and the camera, or poor light, which will inevitably cause a decrease in image quality, resulting in blurred or distorted images, which is not conducive to file retention and comparison. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, apparatus, device and storage medium to restore an image and improve the quality of the image.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, comprising:

[0006] acquiring a first image;

[0007] Restoring the first image based on an image restoration model to obtain a second image, wherein the image restoration model includes a frequency domain processing module and a space domain processing module, the frequency domain processing module is used to process the first image from the frequency domain, and the space domain processing module is used to process the first image from the space domain, the second image is a restored image of the first image, and the quality of the second image is higher than that of the first image;

[0008] The second image is output.

[0009] Optionally, the image restoration model further includes an encoder and a decoder, the frequency domain processing module includes an attention module, and the spatial domain module includes a residual block;

[0010] The restoring the first image based on the image restoration model to obtain the second image includes:

[0011] Performing encoding processing on the first image by the encoder to obtain an encoding result of the first image;

[0012] Processing the encoding result through the attention module to obtain a first feature, and processing the encoding result through the residual block to obtain a second feature;

[0013] Connecting the first feature and the second feature to obtain a connected feature;

[0014] The decoder decodes the connected features to obtain the second image.

[0015] Optionally, the residual block includes a first convolutional layer, a first normalization layer, a first activation layer, a second convolutional layer, and a second normalization layer connected in sequence, and a residual connection is performed between an output end of the second normalization layer and an input end of the first convolutional layer;

[0016] The attention module includes a feature transformation layer, a pooling layer and a channel attention mechanism connected in sequence, the pooling layer includes a first pooling layer and a second pooling layer, the channel attention mechanism includes a multi-layer perceptron MLP and a second activation layer, and the pooling layer is connected to the MLP.

[0017] Optionally, the processing the encoding result by the attention module to obtain the first feature includes:

[0018] Performing Fourier transformation on the encoding result features through the feature transformation layer to obtain transformed features;

[0019] Performing an average pooling operation on the transformed features through the first pooling layer to obtain a first local frequency component, and performing a global average pooling operation on the transformed features through the second pooling layer to obtain a first global frequency component;

[0020] Generate a second local frequency component and a second global frequency component, wherein the second local frequency component is a difference between the transformed feature and the first local frequency component, the frequency of the second local frequency component is higher than the frequency of the first local frequency component, and the second global frequency component is a difference between the transformed feature and the first global frequency component, the frequency of the second global frequency component is higher than the frequency of the first global frequency component;

[0021] The first local frequency component, the second local frequency component, the first global frequency component and the second global frequency component are modulated by the channel attention mechanism to obtain the second feature.

[0022] Optionally, the modulating the first local frequency component, the second local frequency component, the first global frequency component, and the second global frequency component through the channel attention mechanism to obtain the second feature includes:

[0023] Performing weight prediction on the first local frequency component, the second local frequency component, the first global frequency component, and the second global frequency component through the MLP to obtain a first channel weight corresponding to the first local frequency component, a second channel weight corresponding to the second local frequency component, a third channel weight corresponding to the first global frequency component, and a fourth channel weight corresponding to the second global frequency component;

[0024] The first channel weight, the second channel weight, the third channel weight and the fourth channel weight are processed by the second activation layer to obtain a first attention weight corresponding to the first local frequency component, a second attention weight corresponding to the second local frequency component, a third attention weight corresponding to the first global frequency component and a fourth attention weight corresponding to the second global frequency component;

[0025] The second feature is determined based on the first local frequency component, the first attention weight, the second local frequency component, the second attention weight, the first global frequency component, the third attention weight, the second global frequency component and the fourth attention weight.

[0026] Optionally, the loss function of the spatial domain processing module includes an absolute error loss function and a structural similarity loss function, and the loss function of the frequency domain processing module is a focal frequency loss function.

[0027] Optionally, the method further comprises:

[0028] Constructing a loss function of the image restoration model based on the absolute error loss function, the structural similarity loss function and the focal frequency loss function;

[0029] The model is trained based on the loss function through an adaptive moment estimation Adam optimizer, and the image restoration model is obtained after the training.

[0030] In a second aspect, an embodiment of the present application provides an image processing device, including:

[0031] An image acquisition module, used for acquiring a first image;

[0032] an image restoration module, configured to perform restoration processing on the first image based on an image restoration model to obtain a second image, wherein the image restoration model includes a frequency domain processing module and a space domain processing module, wherein the frequency domain processing module is configured to process the first image from a frequency domain, and the space domain processing module is configured to process the first image from a space domain, wherein the second image is a restored image of the first image, and the quality of the second image is higher than that of the first image;

[0033] An image output module is used to output the second image.

[0034] In a third aspect, an embodiment of the present application provides an electronic device, the device comprising: a processor, a memory, and a system bus;

[0035] The processor and the memory are connected via the system bus;

[0036] The memory is used to store a program, wherein the program includes instructions, and when the instructions are executed by the processor, the processor executes any implementation step of the above-mentioned image processing method based on weak measurement technology.

[0037] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on an electronic device, the electronic device executes any implementation step of the above-mentioned image processing method based on weak measurement technology.

[0038] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0039] In an embodiment of the present application, after acquiring the first image, the first image can be restored based on the image restoration model to obtain a second image. The image restoration model may include a frequency domain processing module and a spatial domain processing module, the frequency domain processing module is used to process the first image from the frequency domain, and the spatial domain processing module is used to process the first image from the spatial domain. Then, the second image can be output, and the second image is a restored image of the first image, and the quality of the second image is higher than the quality of the first image. In this way, the first image can be processed from the perspectives of the spatial domain and the frequency domain respectively with the help of the image restoration model, and the frequency domain features can be dynamically learned while mining the spatial domain features to make up for the problem of loss of image information caused by restoration with a single angle model, so that the first image can be effectively restored to obtain the second image, thereby achieving the effect of improving the image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1a An exemplary flowchart of an image processing method provided in an embodiment of the present application;

[0041] Figure 1b Another exemplary flowchart of an image processing method provided in an embodiment of the present application;

[0042] Figure 2 A schematic diagram of the structure of an image restoration model provided in an embodiment of the present application;

[0043] Figure 3 A schematic diagram of the structure of a residual block provided in an embodiment of the present application;

[0044] Figure 4 A schematic diagram of the structure of an attention module provided in an embodiment of the present application;

[0045] Figure 5 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] As mentioned above, in the process of image acquisition by these devices, the imaging system will be affected by various factors, such as sensor noise, relative movement between the object and the camera, or poor light, which will inevitably cause the image quality to deteriorate, resulting in blur or distortion of the image, which is not conducive to the preservation and comparison of files. Based on this, in practical applications, low-quality images can be processed by image restoration. Image restoration aims to reconstruct or restore degraded low-quality images to clear images, while removing image noise and retaining image content and detail information.

[0047] Accordingly, with the rapid development of deep learning in the field of image processing, image restoration methods based on deep learning have become a hot topic. Specifically, there are currently two main solutions for image restoration based on deep learning methods, one is a method based on convolutional neural networks, and the other is a method based on vision transformers (ViT). However, from the perspective of the frequency domain, the models in the above two methods tend to learn specific frequency components, which will lead to the loss of image-specific semantic information and details. Moreover, as the depth of the network increases, although the high-frequency information learning ability of the convolutional neural network has been improved, ViT will still reduce the learning of high-frequency components, which will still lead to poor image restoration effects.

[0048] Based on this, in order to solve the above problem, an embodiment of the present application provides an image processing method, which may include: after acquiring a first image, the first image may be restored based on an image restoration model to obtain a second image. The image restoration model may include a frequency domain processing module and a spatial domain processing module, the frequency domain processing module is used to process the first image from the frequency domain, and the spatial domain processing module is used to process the first image from the spatial domain. Then, a second image may be output, the second image being a restored image of the first image, and the quality of the second image is higher than the quality of the first image.

[0049] In this way, with the help of the image restoration model, the first image can be processed from the perspectives of the spatial domain and the frequency domain respectively, and the frequency domain features can be dynamically learned while mining the spatial domain features to make up for the problem of image information loss caused by restoration with a single angle model. In this way, the first image can be effectively restored to obtain the second image, thereby achieving the effect of improving the image quality.

[0050] It should be noted that the embodiments of the present application may not limit the execution subject of the image processing method. For example, the image processing method of the embodiments of the present application may be applied to data processing devices such as terminal devices or servers. The terminal device may be an electronic device such as a smart phone, a computer, a personal digital assistant (PDA), a tablet computer, etc. The server may be an independent server, a cluster server, or a cloud server.

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0052] Figure 1a A flowchart of an image processing method provided in an embodiment of the present application. Figure 1a As shown, the image processing method provided in the embodiment of the present application takes the corresponding image processing device as the execution subject to describe the specific implementation of the solution. The image processing method may include the following steps S101 to S103.

[0053] S101: Acquire a first image.

[0054] The first image refers to the image to be restored. Taking bank counter services as an example, the first image may be a blurred image corresponding to a scanned business voucher or customer identity information or other documents. Accordingly, this embodiment may not specifically limit the method for obtaining the first image. For example, the image processing device may receive an image obtained by scanning a relevant document using a device other than the counter, and use it as the first image. For another example, the image processing device may first obtain a database that stores images obtained by scanning relevant documents using a device other than the counter, and then the image processing device may obtain the first image from the database.

[0055] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0056] S102: Restoring the first image based on an image restoration model to obtain a second image, wherein the image restoration model includes a frequency domain processing module and a space domain processing module, the frequency domain processing module is used to process the first image from the frequency domain, and the space domain processing module is used to process the first image from the space domain, the second image is a restored image of the first image, and the quality of the second image is higher than that of the first image.

[0057] The image restoration model refers to a model used to restore an image. The following is an exemplary description of the process of obtaining the image restoration model.

[0058] As a possible implementation, combining Figure 1b As shown, the first is the data preprocessing stage, that is, obtaining the training data set. In specific implementation, multiple blurred images collected by devices outside the counter can be obtained, and a clear real image corresponding to each blurred image can be obtained. Among them, each blurred image and its corresponding real image can be stored as a group of image pairs, and an initial data set can be constructed based on multiple groups of image pairs. Next, in order to increase the number and diversity of samples in the training data set, thereby further improving the subsequent performance of the model, in an embodiment of the present application, the initial data set can be further preprocessed, such as at least one of the data enhancement methods such as random cropping, rotation scaling and horizontal flipping, so as to obtain an enhanced data set, that is, a training data set.

[0059] Next, combine Figure 1b As shown, in the model training stage, the above training data set can be input into the model to be trained, the model training is performed using a preset loss function, and the model parameters are updated by back propagation until the training is completed to obtain the image restoration model.

[0060] For ease of understanding, the overall structure of the model to be trained, i.e., the image restoration model, is first introduced with reference to the accompanying drawings.

[0061] Combination Figure 2 As shown, the image restoration model may include a converter, an encoder and a decoder. The encoder is used to encode the image to obtain an encoding result, the converter is used to integrate the encoding result, and the decoder is used to decode the integrated encoding result.

[0062] Specifically, as an example, the encoder may provide a downsampling process to extract features of a blurred image. Accordingly, the encoder may include multiple first convolution blocks, such as three first convolution blocks, wherein each first convolution block may include a third convolution layer, a third normalization layer, and a third activation layer, wherein the third normalization layer is, for example, an instance normalization (InstanceNorm) layer, and the activation function used by the third activation layer is a rectified linear unit (Rectified LinearUnit, ReLU) function. Furthermore, the first first convolution block for receiving an input image may perform a feature extraction operation, with a convolution kernel size of 7×7 and a step size of 1; the next two first convolution blocks perform a downsampling operation, with a convolution kernel size of 3×3 and a step size of 2.

[0063] The converter includes a dual-branch module and a fourth convolutional layer. The dual-branch module includes the above-mentioned frequency domain processing module, spatial domain processing module and connection module. The size of the convolution kernel of the fourth convolutional layer is 3×3. Among them, the frequency domain processing module is specifically an attention module, and the spatial domain module is specifically a residual block. The output end of the attention module and the output end of the residual block can be connected at the channel level through the above-mentioned connection module. In addition, the number of the dual-branch modules is multiple, for example, 9 dual-branch modules are connected in series and then connected to the fourth convolutional layer. In this way, global and local feature extraction can be performed through the attention module to obtain different frequency components, and these frequency components are adaptively enriched with weights. The residual connection provided by the residual block can reduce information loss, and the attention module and the residual block are connected through the connection module, so that the features obtained from the spatial domain and the frequency domain can be fused, so as to dynamically learn the frequency domain features while mining the spatial domain features, make up for the problem of image information loss caused by restoration with a single angle model, and effectively improve the image quality.

[0064] The decoder can provide an upsampling process to generate a restored image corresponding to the blurred image according to the features output by the converter. Accordingly, the decoder may include a deconvolution block, a fifth convolution layer, and a fourth activation layer connected in sequence. There are multiple deconvolution blocks, for example, 2, each deconvolution block has a step size of 2, and each includes a deconvolution layer, a fourth normalization layer, and a fifth activation layer. The fourth normalization layer is, for example, an InstanceNorm layer, and the activation function used in the fifth activation layer is a ReLU function. The size of the convolution kernel of the fifth convolution layer is 7×7. The fourth activation layer is a hyperbolic tangent function (Tanh) layer.

[0065] Furthermore, combined with Figure 3As shown, the residual block may include a first convolutional layer, a first normalization layer, a first activation layer, a second convolutional layer, and a second normalization layer connected in sequence, and a residual connection is performed between the output end of the second normalization layer and the input end of the first convolutional layer. The size of the convolution kernel of the first convolutional layer and the second convolutional layer is 3×3, the first normalization layer and the second normalization layer are both InstanceNorm layers, and the activation function used by the first activation layer is the ReLU function. The output and input of the residual block can be fused through the residual connection.

[0066] Combination Figure 4 As shown, the attention module may include a feature transformation layer, a pooling layer and a channel attention mechanism connected in sequence, the pooling layer includes a first pooling layer and a second pooling layer, the channel attention mechanism includes a multilayer perceptron (MLP) and a second activation layer, and the pooling layer is connected to the MLP. Among them, the feature transformation layer is a fast Fourier transform (FFT) layer. The pooling function used by the first pooling layer is the average pooling (AP) function, which is used to calculate the average value of the elements in the feature map within each window and serve as the representation of the features within the window. The convolution kernel size of the first pooling layer is 3×3, that is, the window size is 3×3. The pooling function used by the second pooling layer is the global average pooling (GAP) function, which is used to calculate the average value of all elements in each feature map and serve as the overall representation of the feature map. The MLP includes a first fully connected layer, a sixth activation layer and a second fully connected layer connected in sequence, and the activation function used by the sixth activation layer is the ReLU function. The second activation layer is the Sigmoid layer.

[0067] Furthermore, combined with Figure 1b As shown, in an embodiment of the present application, different loss functions, namely, spatial domain loss and frequency domain loss, can also be set for the spatial domain processing module and the frequency domain processing module to further improve the performance of the trained model. As an example, the loss function of the spatial domain processing module can include an absolute error loss function and a structural similarity loss function. The loss function of the frequency domain processing module can be a focal frequency loss function.

[0068] Specifically, the absolute error loss function can be used to calculate the sum of the absolute values ​​of the difference between the predicted value and the true value. For the restored image and the true image corresponding to the blurred image, the absolute error loss function is used to calculate the sum of the absolute values ​​of the pixel-level differences between the two, so that the degree of restoration of the restored image to the image authenticity can be evaluated at the pixel level. Accordingly, the absolute error loss function can be shown as follows (1):

[0069]

[0070] in, is the absolute error loss, is the restored image, y is the real image corresponding to the restored image, W is the width of the image, H is the height of the image, is the pixel value of the restored image at the pixel coordinate (i, j), and y(i, j) is the pixel value of the real image at the pixel coordinate (i, j).

[0071] The structural similarity loss function can be used to measure the structural similarity between the blurred image and the restored image. Using the structural similarity loss function to optimize the model can promote the restored image to be as close as possible to the blurred image in structure. Accordingly, the structural similarity loss function can be expressed as follows (2):

[0072]

[0073] in, is the structural similarity loss, is the restored image, y is the real image corresponding to the restored image, μ y is the mean of the real image, σ y is the variance of the real image, is the mean of the restored image, is the variance of the restored image, represents the covariance of the real image and the restored image. c1 and c2 are constants used to avoid calculation errors caused by the denominator being 0. Usually c1 is set to (0.01c max ) 2 , c2=(0.03c max ) 2 , c max is a known constant value that is preset.

[0074] The focal frequency loss function can dynamically adjust the model's attention to different frequency parts in the image during the training process. The focal frequency loss function can be regarded as the weighted distance between the real image and the restored image corresponding to the blurred image in the frequency domain, as shown in the following formula (3):

[0075]

[0076] In the above formula, is the focal frequency loss function, is the restored image, y is the real image corresponding to the restored image, W is the width of the image, H is the height of the image, and F y (u,v) is the frequency value of the real image at the spectrum coordinate (u,v), is the frequency value of the restored image at the spectral coordinate (u, v), and w(u, v) is the weight value at the spectral coordinate w(u, v).

[0077] Furthermore, in determining F y (u,v) and In the process, the image (real image and restored image) can be subjected to FFT operation to convert the image from the spatial domain to the frequency domain, thereby obtaining the representation of the image in the frequency domain spectrum coordinates, which can be calculated by the following formula (4):

[0078]

[0079] Among them, F x (u, v) is the frequency value of image x at the spectral coordinate (u, v) after FFT operation, W is the width of the image, H is the height of the image, (p, q) is the pixel coordinate of image x in the spatial domain, f(p, q) is the pixel value of image x at the pixel coordinate (p, q), i is the imaginary unit, e is the Euler number and follows the Euler formula shown in the following formula (5):

[0080] e iθ =cosθ+isinθ (5)

[0081] In the process of determining w(u,v), the following method shown in formula (6) can be used for calculation:

[0082]

[0083] Among them, w(u,v) is the weight value at the spectrum coordinate w(u,v), F y (u,v) is the frequency value of the real image at the spectrum coordinate (u,v), To restore the frequency value of the image at the spectral coordinate (u, v), α is the scaling factor and is set to 1 by default.

[0084] Then, the weight matrix w is uniformly normalized to the range of [0,1], so the weight value of the frequency part of the image that is most difficult to synthesize will be close to 1, and the weight of the frequency part of the image that is easy to synthesize will be close to 0. In this way, the weight matrix w can be used to adaptively adjust the model's attention to different frequency components in the image, so that balance can be achieved in each frequency component.

[0085] Based on the above three loss functions, in an embodiment of the present application, the model training stage can specifically first construct the loss function of the image restoration model based on the absolute error loss function, the structural similarity loss function and the focal frequency loss function, and then use the Adaptive Moment Estimation (Adam) optimizer to perform model training based on the loss function, and obtain the image restoration model after the training.

[0086] Specifically, the loss function of the image restoration model is shown in the following formula (7):

[0087] L total =λ1L1+λ2L SSIM +λ3L FFL (7)

[0088] Among them, L total is the loss function of the image restoration model, L1 is the absolute error loss function, L SSIM is the structural similarity loss function, L FFL is the focal frequency loss function, λ1 is a hyperparameter controlling the weight of L1, which is set to 1 by default, and λ2 is a hyperparameter controlling L SSIM The default setting is 1, and λ3 is used to control L FFL The default setting is 0.1.

[0089] Moreover, when using the Adam optimizer for model training, the optimizer's learning rate can include two types, namely a and b. Where a = 2e-4, b = [0.5, 0.999]. In addition, taking the case where the model is trained for 200 rounds as an example, the above learning rate is fixed in the first 100 rounds and linearly decays to zero in the next 100 rounds.

[0090] In the embodiment of the present application, Figure 1b As shown in FIG. 1 , after the image restoration model is obtained by model training based on the above model structure and loss function, the blurred image to be restored can be input into the trained image restoration model in the prediction stage to obtain the restored image output by the image restoration model. Accordingly, in order to verify the effect of the restored image, the commonly used indicators in the field of image restoration can be used for evaluation, such as at least one of the mean absolute error, peak signal-to-noise ratio and structural similarity.

[0091] Furthermore, after the image restoration model is obtained through model training based on the above model structure and loss function, the image restoration model can be used to restore the first image. Figure 2As shown, for the above step S102, in a specific implementation, first, the first image can be encoded by the above encoder to obtain an encoding result of the first image; then, the encoding result can be processed by the above attention module to obtain a first feature, and the encoding result can be processed by the above residual block to obtain a second feature; then, the first feature and the second feature can be connected by the above connection module to obtain a connected feature, and finally the connected feature can be decoded by the above decoder to obtain a second image.

[0092] Specifically, as an example, combined with Figure 4 As shown, for the implementation process of obtaining the first feature, first, the encoding result feature can be Fourier transformed through the feature transformation layer to obtain the transformed feature. The above Fourier transformation process can be implemented according to the above formula (4) and formula (5), which will not be repeated here.

[0093] Next, the transformed features can be subjected to an AP operation through the first pooling layer to obtain a first local frequency component, and the transformed features can be subjected to a GAP operation through the second pooling layer to obtain a first global frequency component. Here, the first local frequency component is a lower local frequency component, and the first global frequency component is a lower global frequency component.

[0094] Furthermore, the transformed feature can be used to subtract the first local frequency component to generate a second local frequency component, and the transformed feature can be used to subtract the first global frequency component to generate a second global frequency component, that is, the second local frequency component is the difference between the transformed feature and the first local frequency component, and the frequency of the second local frequency component is higher than the frequency of the first local frequency component. Here, the second local frequency component is a higher local frequency component, and its frequency is higher than the first local frequency component; the second global frequency component is a higher global frequency component, and its frequency is higher than the first global frequency component. For ease of understanding, the generation process of the second local frequency component and the second global frequency component can be introduced in combination with formula (8) and formula (9):

[0095]

[0096] in, is the first global frequency component, GAP is the global average pooling operation, F is the transformed feature, is the second global frequency component.

[0097]

[0098] in, is the first local frequency component, AP is the average pooling operation, F is the transformed feature, is the second local frequency component.

[0099] Then, the first local frequency component, the second local frequency component, the first global frequency component and the second global frequency component are modulated through the above-mentioned channel attention mechanism to obtain the second feature.

[0100] Specifically, for the acquisition process of the second feature, the weight prediction of the first local frequency component, the second local frequency component, the first global frequency component and the second global frequency component can be performed by the above-mentioned MLP to obtain the first channel weight corresponding to the first local frequency component, the second channel weight corresponding to the second local frequency component, the third channel weight corresponding to the first global frequency component and the fourth channel weight corresponding to the second global frequency component, and then the first channel weight, the second channel weight, the third channel weight and the fourth channel weight are processed by the second activation layer to obtain the first attention weight corresponding to the first local frequency component, the second attention weight corresponding to the second local frequency component, the third attention weight corresponding to the first global frequency component and the fourth attention weight corresponding to the second global frequency component. This process can be shown in the following formula (10):

[0101]

[0102] in, is the third attention weight corresponding to the first global frequency component, sigmoid is the S-shaped growth curve, MLP is the multi-layer perceptron, is the first global frequency component.

[0103] Similarly, the first attention weight corresponding to the first local frequency component The second attention weight corresponding to the second local frequency component The fourth attention weight corresponding to the second global frequency component The calculation process of is similar to that of formula (10) above and will not be repeated here.

[0104] Next, the second feature can be determined based on the first local frequency component, the first attention weight, the second local frequency component, the second attention weight, the first global frequency component, the third attention weight, the second global frequency component and the fourth attention weight. The second feature can be represented in the form of a channel feature map, and the determination process can be shown in the following formula (11):

[0105]

[0106] Among them, F q (F) is the second feature, is the third attention weight, is the first global frequency component, is the fourth attention weight, is the second global frequency component, is the first attention weight, is the first local frequency component, is the second attention weight, is the second local frequency component.

[0107] In this way, the attention module performs channel-level attention on global features and local features from the frequency domain level, which enables the network to focus on channels that are helpful for the current task and suppress channels that are irrelevant to the task, thereby improving the model's synthesis efficiency for important areas in blurred images and improving the overall quality of synthesized clear images.

[0108] S103: Output a second image.

[0109] Based on the relevant contents of the above steps S101 to S103, it can be known that in an embodiment of the present application, after acquiring the first image, the first image can be restored based on the image restoration model to obtain a second image. Among them, the image restoration model may include a frequency domain processing module and a spatial domain processing module, the frequency domain processing module is used to process the first image from the frequency domain, and the spatial domain processing module is used to process the first image from the spatial domain. Then, the second image can be output, and the second image is a restored image of the first image, and the quality of the second image is higher than the quality of the first image. In this way, with the help of the image restoration model, the first image can be processed from the perspectives of the spatial domain and the frequency domain respectively, and the frequency domain features can be dynamically learned while mining the spatial domain features, so as to make up for the problem of loss of image information caused by restoration with a single angle model, so that the first image can be effectively restored to obtain the second image, thereby achieving the effect of improving the image quality.

[0110] Furthermore, based on the image processing method provided in the above embodiment, the embodiment of the present application may also provide an image processing device. The image processing device is described below in conjunction with the embodiments and drawings.

[0111] Figure 5 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application. Figure 5 As shown, the image processing device 500 provided in the embodiment of the present application may include:

[0112] An image acquisition module 501 is used to acquire a first image;

[0113] An image restoration module 502 is used to restore the first image based on an image restoration model to obtain a second image, wherein the image restoration model includes a frequency domain processing module and a space domain processing module, wherein the frequency domain processing module is used to process the first image from a frequency domain, and the space domain processing module is used to process the first image from a space domain, wherein the second image is a restored image of the first image, and the quality of the second image is higher than that of the first image;

[0114] The image output module 503 is configured to output the second image.

[0115] Optionally, the image restoration model further includes an encoder and a decoder, the frequency domain processing module includes an attention module, and the spatial domain module includes a residual block;

[0116] The image restoration module 502 includes:

[0117] an encoding module, configured to perform encoding processing on the first image through the encoder to obtain an encoding result of the first image;

[0118] A feature processing module, configured to process the encoding result through the attention module to obtain a first feature, and process the encoding result through the residual block to obtain a second feature;

[0119] A feature connection module, used for connecting the first feature and the second feature to obtain a connected feature;

[0120] A decoding module is used to decode the connected features through the decoder to obtain the second image.

[0121] Optionally, the feature processing module includes:

[0122] A feature transformation module, used for performing Fourier transformation on the encoding result features through the feature transformation layer to obtain transformed features;

[0123] A pooling module, configured to perform an average pooling operation on the transformed features through the first pooling layer to obtain a first local frequency component, and perform a global average pooling operation on the transformed features through the second pooling layer to obtain a first global frequency component;

[0124] A frequency component generating module, used to generate a second local frequency component and a second global frequency component, wherein the second local frequency component is a difference between the transformed feature and the first local frequency component, and the frequency of the second local frequency component is higher than the frequency of the first local frequency component, and the second global frequency component is a difference between the transformed feature and the first global frequency component, and the frequency of the second global frequency component is higher than the frequency of the first global frequency component;

[0125] A modulation module is used to modulate the first local frequency component, the second local frequency component, the first global frequency component and the second global frequency component through the channel attention mechanism to obtain the second feature.

[0126] Optionally, the modulation module is specifically used to:

[0127] Performing weight prediction on the first local frequency component, the second local frequency component, the first global frequency component, and the second global frequency component through the MLP to obtain a first channel weight corresponding to the first local frequency component, a second channel weight corresponding to the second local frequency component, a third channel weight corresponding to the first global frequency component, and a fourth channel weight corresponding to the second global frequency component;

[0128] The first channel weight, the second channel weight, the third channel weight and the fourth channel weight are processed by the second activation layer to obtain a first attention weight corresponding to the first local frequency component, a second attention weight corresponding to the second local frequency component, a third attention weight corresponding to the first global frequency component and a fourth attention weight corresponding to the second global frequency component;

[0129] The second feature is determined based on the first local frequency component, the first attention weight, the second local frequency component, the second attention weight, the first global frequency component, the third attention weight, the second global frequency component and the fourth attention weight.

[0130] Optionally, the image processing device 500 further includes:

[0131] A loss function construction module, used to construct a loss function of the image restoration model based on the absolute error loss function, the structural similarity loss function and the focal frequency loss function;

[0132] The model training module is used to perform model training based on the loss function through an adaptive moment estimation Adam optimizer, and obtain the image restoration model after the training is completed.

[0133] Furthermore, an embodiment of the present application also provides an electronic device, comprising: a processor, a memory, and a system bus;

[0134] The processor and the memory are connected via the system bus;

[0135] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor executes any implementation step of the above-mentioned image processing method.

[0136] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on an electronic device, any step of the above-mentioned image processing method is implemented.

[0137] It can be known from the description of the above implementation mode that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus the necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium, such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in each embodiment of the present application or certain parts of the embodiments. It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other.

[0138] As for the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0139] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0140] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image processing method, characterized in that: include: acquiring a first image; Restoring the first image based on an image restoration model to obtain a second image, wherein the image restoration model includes a frequency domain processing module and a space domain processing module, the frequency domain processing module is used to process the first image from the frequency domain, and the space domain processing module is used to process the first image from the space domain, the second image is a restored image of the first image, and the quality of the second image is higher than that of the first image; The second image is output.

2. The image processing method according to claim 1, characterized in that: The image restoration model further includes an encoder and a decoder, the frequency domain processing module includes an attention module, and the spatial domain module includes a residual block; The restoring the first image based on the image restoration model to obtain the second image includes: Performing encoding processing on the first image by the encoder to obtain an encoding result of the first image; Processing the encoding result through the attention module to obtain a first feature, and processing the encoding result through the residual block to obtain a second feature; Connecting the first feature and the second feature to obtain a connected feature; The decoder decodes the connected features to obtain the second image.

3. The image processing method according to claim 2, characterized in that: The residual block includes a first convolutional layer, a first normalization layer, a first activation layer, a second convolutional layer, and a second normalization layer connected in sequence, and a residual connection is performed between an output end of the second normalization layer and an input end of the first convolutional layer; The attention module includes a feature transformation layer, a pooling layer and a channel attention mechanism connected in sequence, the pooling layer includes a first pooling layer and a second pooling layer, the channel attention mechanism includes a multi-layer perceptron MLP and a second activation layer, and the pooling layer is connected to the MLP.

4. The image processing method according to claim 3, characterized in that: The step of processing the encoding result by the attention module to obtain a first feature includes: Performing Fourier transformation on the encoding result features through the feature transformation layer to obtain transformed features; Performing an average pooling operation on the transformed features through the first pooling layer to obtain a first local frequency component, and performing a global average pooling operation on the transformed features through the second pooling layer to obtain a first global frequency component; Generate a second local frequency component and a second global frequency component, wherein the second local frequency component is a difference between the transformed feature and the first local frequency component, the frequency of the second local frequency component is higher than the frequency of the first local frequency component, and the second global frequency component is a difference between the transformed feature and the first global frequency component, the frequency of the second global frequency component is higher than the frequency of the first global frequency component; The first local frequency component, the second local frequency component, the first global frequency component and the second global frequency component are modulated by the channel attention mechanism to obtain the second feature.

5. The image processing method according to claim 4, characterized in that: The step of performing modulation processing on the first local frequency component, the second local frequency component, the first global frequency component, and the second global frequency component through the channel attention mechanism to obtain the second feature includes: Performing weight prediction on the first local frequency component, the second local frequency component, the first global frequency component, and the second global frequency component through the MLP to obtain a first channel weight corresponding to the first local frequency component, a second channel weight corresponding to the second local frequency component, a third channel weight corresponding to the first global frequency component, and a fourth channel weight corresponding to the second global frequency component; The first channel weight, the second channel weight, the third channel weight and the fourth channel weight are processed by the second activation layer to obtain a first attention weight corresponding to the first local frequency component, a second attention weight corresponding to the second local frequency component, a third attention weight corresponding to the first global frequency component and a fourth attention weight corresponding to the second global frequency component; The second feature is determined based on the first local frequency component, the first attention weight, the second local frequency component, the second attention weight, the first global frequency component, the third attention weight, the second global frequency component and the fourth attention weight.

6. The image processing method according to claim 1, characterized in that: The loss function of the spatial domain processing module includes an absolute error loss function and a structural similarity loss function, and the loss function of the frequency domain processing module is a focal frequency loss function.

7. The image processing method according to claim 6, characterized in that: The method further comprises: Constructing a loss function of the image restoration model based on the absolute error loss function, the structural similarity loss function and the focal frequency loss function; The model is trained based on the loss function through an adaptive moment estimation Adam optimizer, and the image restoration model is obtained after the training.

8. An image processing device, characterized in that: include: An image acquisition module, used for acquiring a first image; an image restoration module, configured to perform restoration processing on the first image based on an image restoration model to obtain a second image, wherein the image restoration model includes a frequency domain processing module and a space domain processing module, wherein the frequency domain processing module is configured to process the first image from a frequency domain, and the space domain processing module is configured to process the first image from a space domain, wherein the second image is a restored image of the first image, and the quality of the second image is higher than that of the first image; An image output module is used to output the second image.

9. An electronic device, characterized in that: The device includes: a processor, a memory, and a system bus; The processor and the memory are connected via the system bus; The memory is used to store a program, wherein the program includes instructions, and when the instructions are executed by the processor, the processor executes the steps of the image processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and when the computer program is executed by a terminal device, the steps of the image processing method according to any one of claims 1 to 7 are implemented.

Images