Image restoration method based on multi-scale component

Abstract

The invention relates to an image restoration method based on multi-scale components. According to the method, low-frequency component enhancement processing is carried out on an original image, then different Gaussian blur kernel parameters are determined according to the original image and an enhanced image, and corresponding Gaussian blur kernels are constructed, so that respective multi-scale low-frequency components of the original image and the enhanced image are obtained; and corresponding multi-scale high-frequency components are extracted and combined, and finally a restored image is obtained based on a high-frequency restoration technology. Through adaptive setting of Gaussian blur kernel parameters, extraction of frequency information of the image under different scales is more accurate, detail features which are easy to omit in a traditional method can be captured, and the texture of the repaired image is clearer and more natural.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to an image restoration method based on multi-scale components. Background Art

[0002] Traditional image restoration methods, such as those based on partial differential equations, use mathematical models to describe diffusion for restoration, but are computationally cumbersome and are powerless when faced with large-area defects or complex textures. Methods based on texture synthesis are good at extracting similar textures for filling, but are slightly inferior for structural restoration. Sample-based methods rely on matching a large number of samples, are easily limited by sample limitations, introduce discordant elements, and distort the image.

[0003] Multi-scale analysis can decompose images into components of different scales through wavelet transform, Gaussian pyramid and other technologies, with low-frequency contours, noise removal and strong contrast, and high-frequency details and realistic effects. However, existing multi-scale restoration methods have hidden shortcomings, especially in parameter settings.

[0004] There is an image restoration method in the prior art that is: perform low-frequency enhancement processing on the original image to be restored to generate a low-frequency enhanced image; use a preset Gaussian blur kernel to process the image to be restored and the low-frequency enhanced image respectively to extract multi-scale low-frequency component information; extract multi-scale high-frequency component information based on the low-frequency component information, and restore the high-frequency component to finally obtain the restored image. However, in terms of setting the parameters of the Gaussian blur kernel, most of them are currently fixed values ​​or rely on manual experience. When encountering an image with concentrated low-frequency information and a narrow frequency band, a large-parameter blur kernel will erase key details, making the restoration result blurry and difficult to distinguish; when encountering an image with complex high-frequency components and a dispersed spectrum, a small parameter is unable to remove noise, affecting the overall effect. Summary of the invention

[0005] Based on this, it is necessary to provide an image restoration method based on multi-scale components that can adaptively adjust blur kernel parameters according to image characteristics to address the above technical problems.

[0006] In a first aspect, the present application provides an image restoration method based on multi-scale components, comprising:

[0007] Perform low-frequency component enhancement processing on the original image to obtain an enhanced image;

[0008] Determine a first Gaussian blur kernel parameter according to the original image, and determine a second Gaussian blur kernel parameter according to the enhanced image;

[0009] A first Gaussian blur kernel is constructed according to the first Gaussian blur kernel parameter, and a second Gaussian blur kernel is constructed according to the second Gaussian blur kernel parameter;

[0010] The original image is processed based on the first Gaussian blur kernel to obtain a multi-scale low-frequency component of the original image; the enhanced image is processed based on the second Gaussian blur kernel to obtain a multi-scale low-frequency component of the enhanced image;

[0011] According to the multi-scale low-frequency components of the original image, the high-frequency components of the original image are extracted to obtain the multi-scale high-frequency components of the original image; according to the multi-scale low-frequency components of the enhanced image, the high-frequency components of the enhanced image are extracted to obtain the multi-scale high-frequency components of the enhanced image;

[0012] Combine the multi-scale high-frequency components of the original image and the multi-scale high-frequency components of the enhanced image to obtain multi-scale combined high-frequency component information;

[0013] According to the high-frequency restoration technology, the multi-scale combined high-frequency component information is subjected to image restoration processing to obtain a restored image.

[0014] In a second aspect, the present application also provides an image restoration device based on multi-scale components, comprising:

[0015] An image low-frequency enhancement module is used to enhance the low-frequency components of the original image to obtain an enhanced image;

[0016] A Gaussian blur kernel parameter determination module, used to determine a first Gaussian blur kernel parameter according to the original image, and to determine a second Gaussian blur kernel parameter according to the enhanced image;

[0017] A Gaussian blur kernel construction module, used to construct a first Gaussian blur kernel according to the first Gaussian blur kernel parameters, and to construct a second Gaussian blur kernel according to the second Gaussian blur kernel parameters;

[0018] The image multi-scale low-frequency component acquisition module is used to process the original image based on the first Gaussian blur kernel to obtain the multi-scale low-frequency component of the original image; and process the enhanced image based on the second Gaussian blur kernel to obtain the multi-scale low-frequency component of the enhanced image;

[0019] The image multi-scale high-frequency component acquisition module is used to extract the high-frequency component of the original image according to the multi-scale low-frequency component of the original image to obtain the multi-scale high-frequency component of the original image; and to extract the high-frequency component of the enhanced image according to the multi-scale low-frequency component of the enhanced image to obtain the multi-scale high-frequency component of the enhanced image;

[0020] An image multi-scale high-frequency component combination module is used to combine the original image multi-scale high-frequency component and the enhanced image multi-scale high-frequency component to obtain multi-scale combined high-frequency component information;

[0021] The image high-frequency restoration module is used to perform image restoration processing on multi-scale combined high-frequency component information according to the high-frequency restoration technology to obtain a restored image.

[0022] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, an image restoration method based on multi-scale components as in the first aspect is implemented.

[0023] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the image restoration method based on multi-scale components as in the first aspect is implemented.

[0024] The above-mentioned image restoration method based on multi-scale components first enhances the low-frequency components of the original image, then determines different Gaussian blur kernel parameters based on the original image and the enhanced image and constructs the corresponding Gaussian blur kernel, thereby obtaining the multi-scale low-frequency components of the original image and the enhanced image, and then extracts and combines the corresponding multi-scale high-frequency components, and finally obtains the restored image based on the high-frequency restoration technology. The adaptive setting of the Gaussian blur kernel parameters makes the frequency information extraction of the image at different scales more accurate, and can capture the detail features that are easily missed by traditional methods, making the texture of the restored image clearer and more natural. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 A schematic diagram of a process of an image restoration method based on multi-scale components provided by the present invention;

[0027] Figure 2 A schematic structural diagram of an image restoration device based on multi-scale components provided by the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0029] refer to Figure 1 , which shows a flow chart of a multi-scale component-based image restoration method provided by the present application, the method comprising the following steps:

[0030] S101: Perform low-frequency component enhancement processing on the original image to obtain an enhanced image.

[0031] Specifically, when performing low-frequency component enhancement processing, a variety of methods can be used, such as using a specific filtering algorithm, such as the Butterworth low-pass filter, to set appropriate cutoff frequency parameters, so that the lower frequency parts of the image can be enhanced, and the detail interference caused by the high-frequency parts can be suppressed, thereby highlighting the overall image outline, general shape and other information, and finally obtaining an enhanced image. For example, for a landscape photo, after low-frequency component enhancement processing, the large outlines of mountains, rivers, and the sky will be clearer, and the color transition will be smoother.

[0032] S102: Determine a first Gaussian blur kernel parameter according to the original image, and determine a second Gaussian blur kernel parameter according to the enhanced image.

[0033] Specifically, when determining the parameters of the first Gaussian blur kernel for the original image, the resolution of the original image itself, the complexity of the content and other factors can be comprehensively considered. If the original image has a high resolution and rich content details, a relatively small standard deviation parameter can be set so that the degree of blur will not be excessive, and more original details can be retained to assist in the subsequent construction of a suitable blur kernel; and for the enhanced image, the parameters of the second Gaussian blur kernel are determined. Since it has been processed by low-frequency component enhancement, the features are different from the original image, and can be determined based on the clarity of the contour after enhancement, color distribution and other conditions. For example, if the contour in the enhanced image is more prominent, when determining the parameters, we can consider how to make the blur kernel built based on it better adapt to this change through parameter settings, and further explore its low-frequency feature-related information. The appropriate parameters can be finally determined after multiple experimental comparisons and observations of the impact of different parameters on the subsequent processing effect.

[0034] S103: constructing a first Gaussian blur kernel according to the first Gaussian blur kernel parameters, and constructing a second Gaussian blur kernel according to the second Gaussian blur kernel parameters.

[0035] Specifically, after the parameters of the first Gaussian blur kernel are determined, a mathematical formula is used to construct the first Gaussian blur kernel. The Gaussian blur kernel is essentially a two-dimensional weight matrix, and the values ​​of its elements are calculated based on the Gaussian function and the set parameters (such as standard deviation, etc.). These weight values ​​can reflect the degree of influence when blurring pixels at different positions in the image. Similarly, for the second Gaussian blur kernel, according to its corresponding parameters, the corresponding two-dimensional weight matrix is ​​constructed by the same principle, so that they can be used to operate on the image pixels according to the corresponding weights in the future to achieve a specific blur effect, thereby separating low-frequency components of different scales.

[0036] S104: Processing the original image based on the first Gaussian blur kernel to obtain multi-scale low-frequency components of the original image; processing the enhanced image based on the second Gaussian blur kernel to obtain multi-scale low-frequency components of the enhanced image.

[0037] Specifically, when the first Gaussian blur kernel is used to process the original image, the center of the blur kernel is aligned with each pixel of the original image in turn, and then the weighted sum operation is performed on the pixel and its surrounding pixels according to the weights corresponding to the elements in the blur kernel. By changing the scale of the blur kernel (such as changing its size or other related parameters), the low-frequency components of the original image at different scales can be obtained, and these low-frequency components are combined to obtain the multi-scale low-frequency components of the original image. Similarly, when the second Gaussian blur kernel is used to process the enhanced image, the operation process is similar, but based on the characteristics of the enhanced image itself and the corresponding parameters of the second Gaussian blur kernel, the multi-scale low-frequency components of the enhanced image are finally obtained. These low-frequency components can reflect the general outline of the enhanced image at different scales, the characteristics of gentle changes, and other information.

[0038] S105: extracting high-frequency components of the original image according to the multi-scale low-frequency components of the original image to obtain multi-scale high-frequency components of the original image; extracting high-frequency components of the enhanced image according to the multi-scale low-frequency components of the enhanced image to obtain multi-scale high-frequency components of the enhanced image.

[0039] Specifically, the high-frequency components of the original image can be extracted by subtracting the low-frequency components from the overall information of the image. Specifically, for the original image at each scale, the pixel value matrix of the original image itself is subtracted from the pixel value matrix of the corresponding multi-scale low-frequency components. The difference is the high-frequency component at that scale. The high-frequency components at multiple scales constitute the multi-scale high-frequency components of the original image. The high-frequency components reflect the information such as the dramatic changes in details and edges in the image. Similarly, for the enhanced image, in the same way, the pixel value of the enhanced image is subtracted from the corresponding multi-scale low-frequency component pixel value to obtain the multi-scale high-frequency components of the enhanced image. These high-frequency components contain the details and features left after the enhanced image is processed by low frequency.

[0040] S106: Combining the multi-scale high-frequency component of the original image and the multi-scale high-frequency component of the enhanced image to obtain multi-scale combined high-frequency component information.

[0041] Specifically, when combining these two multi-scale high-frequency components, their respective characteristics at different scales and their impact on the final restoration effect can be considered. They can be combined in a certain weight distribution method. For example, based on experiments or experience, the high-frequency components of the original image at certain scales can better reflect the real details of the image, so they are given relatively high weights, while the high-frequency components of the enhanced image at other scales are more advantageous, so their weights are increased accordingly. The high-frequency components of the original image and the enhanced image corresponding to each scale are fused by weighted summation and other methods, and finally multi-scale combined high-frequency component information is formed, which can combine the advantages of both.

[0042] S107: Perform image restoration processing on the multi-scale combined high-frequency component information according to the high-frequency restoration technology to obtain a restored image.

[0043] Specifically, there are many high-frequency restoration technologies. For example, a sparse representation-based method can be used to utilize the sparse characteristics of the image, and the multi-scale combined high-frequency component information is sparsely represented through a suitable dictionary to find the most matching atomic combination to reconstruct the high-frequency part of the image; or a deep learning-based method can be used to input the multi-scale combined high-frequency component information into a trained neural network model, so that the model can learn how to recover complete and reasonable image details from these high-frequency component information. After such processing, the final restored image is obtained, which retains the overall contour and other low-frequency features while the details are reasonably restored and enhanced, making the visual effect clearer and more natural.

[0044] The above-mentioned image restoration method based on multi-scale components first enhances the low-frequency components of the original image, then determines different Gaussian blur kernel parameters based on the original image and the enhanced image and constructs the corresponding Gaussian blur kernel, thereby obtaining the multi-scale low-frequency components of the original image and the enhanced image, and then extracts and combines the corresponding multi-scale high-frequency components, and finally obtains the restored image based on the high-frequency restoration technology. The adaptive setting of the Gaussian blur kernel parameters makes the frequency information extraction of the image at different scales more accurate, and can capture the detail features that are easily missed by traditional methods, making the texture of the restored image clearer and more natural.

[0045] In an optional embodiment, the first Gaussian blur kernel parameter is the standard deviation σ1 of the first Gaussian blur kernel, and the second Gaussian blur kernel parameter is the standard deviation σ2 of the second Gaussian blur kernel; determining the first Gaussian blur kernel parameter according to the original image, and determining the second Gaussian blur kernel parameter according to the enhanced image, comprises the following steps:

[0046] Performing a two-dimensional discrete Fourier transform on the original image to obtain the original image spectrum; performing a two-dimensional discrete Fourier transform on the enhanced image to obtain the enhanced image spectrum;

[0047] The initial range of the standard deviation σ1 is determined according to the distribution of the spectrum energy in the original image spectrum, and the initial range of the standard deviation σ2 is determined according to the distribution of the spectrum energy in the enhanced image spectrum;

[0048] The golden section search method is used to iteratively search for the optimal σ1 value within the initial range of the standard deviation σ1, and the golden section search method is used to iteratively search for the optimal σ2 value within the initial range of the standard deviation σ2.

[0049] Specifically, the teammates perform a two-dimensional discrete Fourier transform:

[0050] (1) For the original image: Two-dimensional discrete Fourier transform (2D-DFT) is a mathematical method to convert the original image from the spatial domain to the frequency domain. Through the two-dimensional discrete Fourier transform, the original image is decomposed into components of different frequencies to form the original image spectrum. In the frequency domain, low-frequency components correspond to the general outline and slowly changing parts of the image, and high-frequency components correspond to the details and edges of the image. For example, after a person's image is transformed, low-frequency information such as facial contours has higher energy near the center of the spectrum, while high-frequency information corresponding to details such as the person's hair and clothing texture is distributed at the edge of the spectrum.

[0051] (2) For the enhanced image: Similarly, the enhanced image is subjected to a two-dimensional discrete Fourier transform to obtain the enhanced image spectrum. Since the enhanced image has been processed by low-frequency component enhancement, its spectrum distribution is different from that of the original image, and the low-frequency energy may be more concentrated or the distribution pattern may change.

[0052] For determining the initial range of standard deviation:

[0053] (1) Based on the original image spectrum: Observe the distribution of spectral energy in the original image spectrum. If the low-frequency energy has a high value in a wide frequency range, it means that the low-frequency components of the image are rich and widely distributed. At this time, the initial range of σ1 can be set relatively large to better extract the low-frequency components; conversely, if the low-frequency energy is concentrated in a smaller frequency range, the initial range can be appropriately reduced.

[0054] (2) Based on the enhanced image spectrum: Similarly, the initial range of σ2 is determined based on the energy distribution of the enhanced image spectrum. If the low-frequency enhancement effect of the enhanced image is significant and the low-frequency energy is more concentrated, the initial range of σ2 can be adjusted accordingly to adapt to the characteristics of the enhanced image.

[0055] For iterative search for the optimal value using the golden section search method:

[0056] (1) Searching for the optimal value of σ1: Within the determined initial range of σ1, the golden section search method approaches the optimal value by continuously iteratively narrowing the search interval. Each iteration selects two points at specific positions in the interval (determined by the golden section ratio), calculates the effect evaluation index of the Gaussian blur kernel corresponding to these two points on the original image after processing (such as the degree of matching with the low-frequency features of the original image, etc.), discards part of the interval based on the evaluation result, and repeats the process until the preset number of iterations or accuracy requirements are reached, and the optimal σ1 value is obtained.

[0057] (2) Searching for the optimal value of σ2: The same golden section search method is used to iterate σ2 within its initial range. By evaluating the processing effect of the Gaussian blur kernel corresponding to different σ2 values ​​on the enhanced image, the optimal value of σ2 that is most suitable for the enhanced image is gradually found.

[0058] Among the parameters of the Gaussian blur kernel, the standard deviation determines the shape and blur degree of the Gaussian blur kernel. A smaller standard deviation produces a weaker blur effect and can retain more image details; a larger standard deviation will make the image more blurred and highlight the overall contour more.

[0059] In an optional embodiment, determining an initial range of the standard deviation σ1 according to the distribution of spectral energy in the original image spectrum, and determining an initial range of the standard deviation σ2 according to the distribution of spectral energy in the enhanced image spectrum, comprises the following steps:

[0060] S1: The original image spectrum or enhanced image spectrum is recorded as F(u,v), and the spectrum amplitude of F(u,v) is calculated. The proportion P of the energy of F(u,v) in the low frequency band to the total energy is calculated according to the following formula:

[0061]

[0062] Among them, Re[F(u,v)] is the real part of F(u,v), Im[F(u,v)] is the imaginary part of F(u,v); the frequency range of the low frequency band is (0,α×f max ], α is the preset magnification, f max is the maximum frequency of F(u,v), u,v∈L means that the values ​​of u and v are in the low frequency band, and u,v∈A means that the values ​​of u and v are in the full frequency band of F(u,v);

[0063] S2: Determine the frequency interval [f1,f2] where the energy peak of F(u,v) is located, including:

[0064] From the spectrum amplitude |F(u,v)|, we can get the maximum spectrum amplitude |F(u max ,v max )|; In F(u,v), the maximum point (u max ,v max ) is the center and the sum of the spectrum amplitudes in the circular area with a radius of r is S(r);

[0065] when When [u max -r,v max -r] to [u max +r,v max +r] as the frequency range corresponding to the preset radius increase, and δ is the preset change rate threshold;

[0066] S3: The ratio P and Input into the trained neural network model, and output a1 and a2, where a1 < a2; if F(u, v) is the spectrum of the original image, the initial range of the standard deviation σ1 is [a1, a2], and if F(u, v) is the spectrum of the enhanced image, the initial range of the standard deviation σ2 is [a1, a2].

[0067] Specifically, for calculating the ratio of the spectrum amplitude to the low-frequency energy:

[0068] (1) Spectrum amplitude calculation: Denote the spectrum of the original image or the enhanced image as F(u, v). Since the spectrum after the two-dimensional discrete Fourier transform is in complex form, to quantify the magnitude of each frequency component in the spectrum, the spectrum amplitude can be calculated. According to the formula By taking the square root of the sum of the squares of the real part Re[F(u, v)] and the imaginary part Im[F(u, v)], the spectrum amplitude is obtained.

[0069] (2) Low-frequency energy ratio calculation: Define the low-frequency band frequency range as (0, α×f max (1), where α is a preset magnification factor that can be set according to requirements; f max is the maximum frequency of F(u, v). Calculate the ratio P of the low-frequency band energy to the total energy. The formula is u, v ∈ L indicates taking values in the low-frequency band, and u, v ∈ A indicates taking values in the full frequency band. The ratio P can reflect the proportion of the low-frequency components of the image in the total spectrum energy. If the proportion of the low-frequency energy of the image is high, such as there are many large smooth areas like the sky and the earth in a landscape image, the value of P will be large.

[0070] For determining the frequency interval where the energy peak is located:

[0071] (1) Finding the maximum value of the spectrum amplitude: Find the maximum value |F(u max , v max )| from the spectrum amplitude |F(u, v)|. The point (u max , v max ) corresponds to the frequency position with the strongest energy in the spectrum and is the key reference point for determining the energy peak interval.

[0072] (2) Determining the frequency interval: Centered on the maximum value point (u max , v max ), calculate the sum S(r) of the spectrum amplitudes within a circular region with a radius of r. As the radius increases from r to r + Δr (Δr is the preset radius increment), when (δ is the preset change rate threshold), it indicates that the main region of energy concentration has been passed. At this time, from [u max -r, v max -r] to [u max +r, v max+r] is determined as the frequency interval [f1, f2] where the energy peak is located.

[0073] For determining the initial range of the standard deviation through a neural network, the training method of the neural network model can be as follows:

[0074] (1) Data collection and preprocessing:

[0075] Collect image data: Build a dataset containing various types of images, including natural scenes, people, buildings, etc., to ensure that the network can learn a wide range of image features.

[0076] Image transformation and annotation: Perform a two-dimensional discrete Fourier transform on the collected images to obtain spectrum information. According to the above method, calculate the spectrum amplitude, low-frequency energy ratio P, and the frequency interval [f1, f2] where the energy peak is located, and then calculate At the same time, according to the image characteristics, the appropriate Gaussian blur kernel standard deviation range is determined as a label by manual annotation or other reliable algorithms.

[0077] (2) Model construction:

[0078] Select the network structure: Multilayer Perceptron (MLP) can be used because it is good at handling complex nonlinear relationships between input features and outputs. The network consists of an input layer, multiple hidden layers, and an output layer. The number of nodes in the input layer corresponds to the number of input features, that is, the low-frequency energy ratio P and There are two nodes in total. The hidden layer can be set to 2-3 layers, and the number of nodes in each layer is between 32-128. The best configuration is determined through experiments. The output layer has two nodes, corresponding to the output a1 and a2 respectively.

[0079] Activation function selection: The hidden layer can use the ReLU activation function to solve the gradient vanishing problem and speed up training. The output layer can use the linear activation function to ensure that the output value is within a reasonable range to represent the initial range of the standard deviation.

[0080] (3) Model training:

[0081] Define the loss function: Use the mean square error (MSE) loss function to calculate the square mean of the difference between the predicted standard deviation range and the labeled true standard deviation range. The formula is Where n is the number of samples, and is the model prediction value, a 1i and a 2i is the true label value.

[0082] Choose an optimizer: You can use the Adam optimizer, which can adaptively adjust the learning rate, speed up convergence in the early stages of training, and avoid excessive oscillation when approaching the optimal solution. The initial learning rate can be set to 0.001, and the learning rate can be adjusted during training according to the training situation.

[0083] Training process: Divide the preprocessed data set into training set, validation set and test set, and the ratio can be set to 7:2:1. During the training process, batches of training data are input into the network, the loss value is calculated, and the network parameters are updated through the back propagation algorithm. After each training round, the model performance is evaluated on the validation set to observe the changes in loss value and accuracy. If the loss value on the validation set does not decrease for multiple consecutive rounds, the training can be stopped in advance to prevent overfitting.

[0084] (4) Model evaluation and tuning:

[0085] Evaluation indicators: Use indicators such as mean absolute error (MAE) and root mean square error (RMSE) to evaluate model performance.

[0086] Model tuning: According to the evaluation results, adjust the network structure (such as the number of hidden layer nodes and layers), optimizer parameters (learning rate, momentum, etc.) or add data enhancement methods (such as rotating and cropping images), and retrain the model until the model achieves satisfactory performance indicators on the test set and can accurately output the initial range of standard deviation suitable for image characteristics.

[0087] In an optional embodiment, performing low-frequency component enhancement processing on the original image to obtain an enhanced image includes the following steps:

[0088] Use the trained low-frequency enhancement network to extract features from the original image to obtain local features and intermediate image features;

[0089] Aggregate the intermediate image features to obtain feature aggregation results;

[0090] Perform global feature extraction on the feature aggregation results to obtain global features;

[0091] Using the feature fusion method based on the attention mechanism, local features and global features are fused to obtain fused features;

[0092] Based on the fused features and the original image, a low-frequency enhanced image is obtained.

[0093] Specifically, the low-frequency enhancement network is used to extract features:

[0094] The low-frequency enhancement network can be a deep neural network model trained with data, which is composed of multiple convolutional layers, pooling layers, and fully connected layers. The convolutional layer slides on the original image through different convolution kernels, which can capture local features of different scales and directions in the image, such as the edges and textures of objects in the image. The pooling layer is used to downsample the feature map, which reduces the amount of data while retaining the main features and improving computational efficiency.

[0095] The original image is input into the above-mentioned low-frequency enhancement network, and the network processes the image according to its preset structure and trained parameters. In this process, the network will output local features and intermediate image features. Local features contain unique information of each local area of ​​the image, such as the branches and leaves of trees and the shape of flowers in a landscape image; intermediate image features are intermediate results generated by the network during the processing process, which integrate a certain range of image information.

[0096] Aggregate intermediate image features:

[0097] There are many ways to aggregate intermediate image features, such as average pooling, maximum pooling, or aggregation methods based on attention mechanisms. Average pooling calculates the average value of elements in a certain area of ​​the feature map to represent the features of the area. It can smooth features and reduce the impact of noise. Maximum pooling selects the maximum value in a certain area as the representative feature. This method can highlight significant features. Aggregation methods based on attention mechanisms assign different weights according to the importance of each feature, and pay more attention to the aggregation of key features.

[0098] By aggregating the intermediate image features, the scattered, high-dimensional intermediate features are transformed into more representative, relatively low-dimensional feature aggregation results. This processing can not only reduce the amount of data and the complexity of subsequent processing, but also integrate information, making the features more compact and representative, which helps to extract higher-level global features.

[0099] Perform global feature extraction on the feature aggregation results:

[0100] You can use methods such as fully connected layers or global average pooling to extract global features from feature aggregation results. The fully connected layer can learn the complex relationship between aggregated features, and output global features that can reflect the overall characteristics of the image by weighted summing of all input features and processing them through activation functions. Global average pooling calculates the average value of the entire feature aggregation result to obtain a vector that can represent the global information of the image, such as the overall tone and general shape of the image.

[0101] Global features can describe the properties of an image from a macroscopic perspective. They integrate information from all parts of the image to understand the overall structure and content of the image. In the case of low-frequency enhancement, global features help grasp the overall outline of the image and large-scale change trends.

[0102] The features are fused using the feature fusion method based on the attention mechanism:

[0103] The feature fusion method based on the attention mechanism can calculate the attention weights for local features and global features respectively. It assigns higher weights to features that contribute more to the final result and lower weights to relatively unimportant features by learning the importance of different features. For example, if the details of a local area in an image are very critical for low-frequency enhancement, then the attention weight corresponding to the local feature will be higher.

[0104] In the fusion process, the local features and global features are weighted and summed according to the calculated attention weights to achieve feature fusion. The fused features contain both rich local detail information of the image and the overall structure and content information, which can better reflect the characteristics of the image.

[0105] Based on the fusion features and the original image, a low-frequency enhanced image is obtained:

[0106] The fused features and the original image can be input into a generation module, which can be composed of a deconvolution layer, an upsampling layer, and a convolution layer. The deconvolution layer and the upsampling layer are used to restore the size of the image, and the convolution layer further fuses and processes the fused features and the information of the original image, and generates a low-frequency enhanced image by adjusting the pixel value of the image.

[0107] The generated low-frequency enhanced image enhances the low-frequency components while retaining the basic information of the original image. The overall outline of the image is clearer, the color transition is smoother, and the large area and slowly changing areas in the image can be highlighted.

[0108] In an optional embodiment, a feature fusion method based on an attention mechanism is used to fuse local features and global features to obtain fused features, including the following steps:

[0109] Reshape the intermediate image features to obtain reshaped image features;

[0110] Project the reshaped intermediate image features into the projection matrix to obtain the projection features;

[0111] Calculate the self-similarity matrix of the projected features;

[0112] Use neural networks to learn the importance weights of local and global features in different image regions based on the self-similarity matrix;

[0113] The local features and global features are fused according to the importance weights to obtain fused features.

[0114] Specifically, the intermediate image features are resized:

[0115] After a series of processing by the network, the size and dimension of the intermediate image features may not meet the requirements of subsequent calculations. Resizing can adjust the shape of the features to adapt them to a specific computing model or algorithm. For example, in many neural network architectures, subsequent computing operations have strict requirements on the size of the input data. By resizing, it can ensure that the data can flow smoothly in the network and facilitate the alignment and comparison of features.

[0116] You can use specialized operation layers in neural networks, such as the view function in PyTorch or the reshape function in TensorFlow. These functions can reshape intermediate image features according to the specified target shape.

[0117] Project the reshaped intermediate image features into the projection matrix:

[0118] The projection operation can map the high-dimensional reshaped intermediate image features into a new feature space, where the features may have better distinguishability and expressiveness. Through projection, the features can be transformed from the original complex representation into a form that is more conducive to calculating self-similarity, which helps to explore the potential relationship between features.

[0119] The projection matrix can be obtained by training a complete model. During the model training process, the model can automatically adjust the parameters of the projection matrix according to the characteristics of the input data and the task requirements, so that the projected features can retain useful information to the greatest extent while reducing noise and redundancy. For example, the linear transformation matrix P can be used to project the reshaped intermediate image features F through matrix multiplication to obtain the projection features F proj =F×P. The dimension of the projection matrix P can be designed according to the dimension of the input features and the target projection space.

[0120] Compute the self-similarity matrix of the projected features:

[0121] The self-similarity matrix is ​​used to measure the similarity between each element in the projection feature. By calculating the self-similarity matrix, the correlation between features at different positions can be obtained, which helps to determine the importance weights of local features and global features in different image regions in the future. If the corresponding element values ​​of the features at two positions in the self-similarity matrix are high, it means that the features at these two positions have high similarity, and their roles in the image may be similar or related.

[0122] The method for calculating the self-similarity matrix can be to use methods such as cosine similarity or dot product. Taking cosine similarity as an example, for the projection feature F proj , the element S of its self-similarity matrix S ij It can be calculated by proj The cosine similarity between the i-th eigenvector and the j-th eigenvector is obtained, that is, in and They represent the i-th and j-th eigenvectors in the projection features respectively.

[0123] Using neural networks to learn importance weights based on the self-similarity matrix:

[0124] The neural network used in this step can be a small network specially designed for learning weights. It takes the self-similarity matrix as input and processes and analyzes the information in the self-similarity matrix through a series of neural network layers, such as fully connected layers and convolutional layers. The goal of this network is to learn the importance weights of local features and global features in different image regions. These weights can reflect the relative importance of different features in the image.

[0125] During the training process, the parameters of the neural network are continuously adjusted through the back-propagation algorithm so that the importance weights of the network output can optimize a specific objective function. For example, the cross entropy loss function or the mean square error loss function can be used to compare the weights of the network output with the pre-labeled true weights (if any) or the ideal weight distribution designed according to the task requirements, and the network parameters are optimized by minimizing the loss function. After multiple iterations of training, the neural network can gradually learn weights that accurately reflect the importance of features.

[0126] The local features and global features are fused according to the importance weights:

[0127] After obtaining the importance weights of local features and global features in different image regions, these weights are applied to the corresponding local features and global features respectively. Specifically, for each image region, the local feature is multiplied by the corresponding local importance weight, the global feature is multiplied by the corresponding global importance weight, and then the two are added to obtain the fused feature. Assume that the local feature is L, the global feature is G, and the local importance weight is ω L , the global importance weight is ω G , then the fusion feature F fusion =ω L ×L+ω G ×G.

[0128] This feature fusion method based on the attention mechanism can dynamically adjust the contribution of local features and global features according to the characteristics of different areas of the image. In areas with rich details in the image, the weight of local features may be higher, thereby highlighting these details; in areas that reflect the overall structure of the image, the weight of global features may be higher, which helps to maintain the overall consistency of the image. The fused features obtained in this way contain rich local information while taking into account the global structure of the image, providing more expressive feature representations for subsequent image processing tasks.

[0129] In an optional embodiment, before processing the original image using the trained low-frequency enhancement network, the following steps are also included:

[0130] Use a sliding window to calculate the local variance of the original image;

[0131] Use the Sobel operator to calculate the horizontal and vertical gradients of the original image;

[0132] Based on the preset rules, the convolution kernel parameters of the low-frequency enhancement network are determined according to the local variance, horizontal gradient and vertical gradient.

[0133] Specifically, a sliding window is used to calculate the local variance of the original image:

[0134] The sliding window is a rectangular area that moves pixel by pixel on the image with a fixed step size. Its size can be set according to the image characteristics and analysis purpose, and can be 3×3, 5×5, etc. Taking a 3×3 window as an example, the window starts from the upper left corner of the image and moves one pixel horizontally or vertically each time.

[0135] At each window position, calculate the variance of the pixel values ​​within the window. Let the pixel values ​​within the window be x1, x2, ..., x n (n is the number of pixels in the window), the mean is Then the local variance The local variance reflects the degree of discreteness of the pixel values ​​within the window. The larger the variance, the greater the difference in pixel values ​​and the richer the details or texture in the area; the smaller the variance, the relatively smoother the area.

[0136] Use the Sobel operator to calculate the horizontal and vertical gradients of the original image:

[0137] The Sobel operator is an edge detection operator that contains two convolution kernels, horizontal and vertical. x Used to detect vertical edges, the vertical convolution kernel K y Used to detect horizontal edges. Taking the 3×3 Sobel convolution kernel as an example,

[0138] Convolve the convolution kernel of the Sobel operator with the original image. For each pixel in the image, take the pixel as the center, cover the convolution kernel on the corresponding area, and perform weighted summation. For example, when calculating the horizontal gradient, K x Multiply and sum the pixel values ​​in the corresponding area of ​​the image to get the horizontal gradient value G of the pixel x ; Let the pixel value of the 3×3 image area centered on pixel (i, j) be Then the horizontal gradient G of the pixel point (i, j) is x The calculation formula for (i, j) is: x (i,j)=(-1)×a 11 +0×a 12 +1×a 13 +(-2)×a 21 +0×a 22 +2×a 23 +(-1)×a 31 +0×a 32 +1×a 33 Similarly, using K y Calculate the vertical gradient value G y The magnitude of the gradient value can reflect the drastic degree of change in the pixel value of the image in that direction. The area with a large gradient value is often the edge of the image.

[0139] Determine the convolution kernel parameters of the low-frequency enhancement network based on preset rules:

[0140] The preset rules can comprehensively consider the local variance, horizontal gradient, and vertical gradient. For example, if the local variance is large and the horizontal and vertical gradients are also large, it means that the area is rich in details and has obvious edges. In order to better preserve the details when enhancing the low-frequency components, the convolution kernel size of the low-frequency enhancement network can be made smaller and the weight distribution can be more focused on the central pixel to reduce excessive smoothing of details. On the contrary, if the local variance is small and the gradient value is small, it means that the area is relatively smooth, and the convolution kernel size can be appropriately increased to enhance the extraction of low-frequency information.

[0141] When determining the convolution kernel parameters, the local variance, horizontal and vertical gradients can be used as input features and input into a pre-designed parameter determination model. The model can be a simple linear regression model or a more complex neural network model. The model learns these features of the image and establishes a mapping relationship with the appropriate convolution kernel parameters, thereby outputting the convolution kernel parameters required by the low-frequency enhancement network, such as the convolution kernel size, weight value, etc., so that the low-frequency enhancement network can perform targeted processing according to the characteristics of different areas of the image.

[0142] In an optional embodiment, according to the high-frequency restoration technology, image restoration processing is performed on the multi-scale combined high-frequency component information to obtain a repaired image, including the following steps:

[0143] The high-frequency components of different scales in the multi-scale combined high-frequency component information are stacked along the channel dimension to form a comprehensive high-frequency component;

[0144] Inputting the integrated high-frequency component into a preset high-frequency recovery model and outputting the Laplace component;

[0145] According to the following formula, based on the mean filtering method, the low-frequency estimation image is obtained by calculating the average value of each pixel in the original image and its neighborhood pixels:

[0146]

[0147] Where I(i+x,j+y) is the pixel value in the filter window of the original image, m and n are the number of rows and columns of the filter window respectively, and I lf (i, j) is the low-frequency estimation image; x and y are the coordinate variables relative to the center pixel in the filter window;

[0148] The Laplace component is added to the low-frequency estimation image pixel by pixel to obtain the repaired image.

[0149] Specifically, a comprehensive high-frequency component is formed: the multi-scale combined high-frequency component information contains high-frequency components at different scales, and the high-frequency components reflect the detailed information of the image at different levels of fineness. In order to process this information in a unified framework later, the high-frequency components are stacked along the channel dimension. Assume that there are k high-frequency components of different scales in the multi-scale combined high-frequency component information, and the size of each high-frequency component is H×W×C i (H is height, W is width, C is i is the number of channels of the corresponding scale), the size of the comprehensive high-frequency component formed after stacking is The high-frequency components of each scale are integrated by stacking, providing comprehensive detail feature input for the high-frequency recovery model, so that the model can simultaneously use information of different scales for image restoration.

[0150] Input the high-frequency restoration model to obtain the Laplace component: Input the comprehensive high-frequency component into the preset high-frequency restoration model, which can be a trained network model, and its structure can be based on a convolutional neural network (CNN), a generative adversarial network (GAN) or other architectures suitable for image restoration tasks. The model training process can use image samples containing high-frequency details, and by adjusting the network parameters, learn how to restore the missing or damaged high-frequency parts of the image from the comprehensive high-frequency components, and finally output the Laplace component. The Laplace component contains the parts of the image that are rich in details and change dramatically, such as the edges, textures, and subtle features of objects in the image.

[0151] Calculate low-frequency estimation image: The mean filter method is used to calculate the low-frequency estimation image. The mean filter is implemented by calculating the average value of each pixel in the original image and its neighboring pixels. The size of the filter window is determined by the row m and column n. For each pixel point (i, j) in the original image, the filter window is centered on the pixel. Within the window, the value range of the coordinate variable x is from arrive The value range of y is from arrive By summing all pixel values ​​I(i+x,j+y) in the window and dividing it by the total number of pixels in the window m×n, the value I of the low-frequency estimated image at the pixel point (i,j) is obtained. lf (i, j). Mean filtering can smooth the image, remove high-frequency noise in the image, and highlight the low-frequency components of the image, that is, the rough outline and slowly changing areas.

[0152] Obtain the repaired image: add the Laplace component output by the high-frequency restoration model to the calculated low-frequency estimation image pixel by pixel. Since the Laplace component contains the high-frequency details of the image, and the low-frequency estimation image represents the general outline and low-frequency information of the image, the addition of the two can restore the detailed information of the image while retaining the overall structure of the image, thereby obtaining the repaired image. In this way, the repaired image has a clear outline and is reasonably supplemented and enhanced in the details, which improves the image quality and visual effect, and achieves the purpose of image restoration.

[0153] The above-mentioned image restoration method based on multi-scale components determines the convolution kernel parameters of the low-frequency enhancement network by calculating the local variance of the original image and the gradient of the Sobel operator, and uses the low-frequency enhancement network to enhance the low-frequency components of the original image to obtain an enhanced image; performs a two-dimensional discrete Fourier transform on the original and enhanced images, calculates the low-frequency energy ratio of the two images according to the spectrum energy distribution, determines the energy peak frequency range of the two images, inputs the low-frequency energy ratio and the energy peak frequency range into the neural network, outputs the initial range of the corresponding Gaussian blur kernel standard deviation, and determines the optimal standard deviation within the initial range using the golden section search method; constructs the corresponding Gaussian blur kernel according to the optimal standard deviation corresponding to the original image and the enhanced image, thereby obtaining the multi-scale low-frequency components of the original image and the enhanced image, and then extracts and combines the corresponding multi-scale high-frequency components therefrom, and fuses the combined high-frequency components with the low-frequency estimation image through the high-frequency recovery technology to obtain the restored image. The standard deviation of the Gaussian blur kernel is determined based on image characteristics, so that the blur kernel can accurately adapt to image features. In the process of multi-scale component extraction, the high- and low-frequency information of the image can be more effectively separated. Therefore, during image restoration, details can be better preserved and contours can be highlighted, greatly improving the quality and clarity of the restored image and improving the visual effect.

[0154] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0155] Based on the same inventive concept, the embodiment of the present application also provides a device for implementing the above-mentioned method for image restoration based on multi-scale components. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in the embodiments of one or more image restoration devices based on multi-scale components provided below can refer to the above-mentioned limitations on a method for image restoration based on multi-scale components, and will not be repeated here.

[0156] In an exemplary embodiment, Figure 2 As shown, a multi-scale component-based image restoration device 20 is provided, comprising:

[0157] The image low-frequency enhancement module 21 is used to perform low-frequency component enhancement processing on the original image to obtain an enhanced image.

[0158] The Gaussian blur kernel parameter determination module 22 is used to determine the first Gaussian blur kernel parameter according to the original image, and to determine the second Gaussian blur kernel parameter according to the enhanced image.

[0159] The Gaussian blur kernel construction module 23 is used to construct a first Gaussian blur kernel according to the first Gaussian blur kernel parameters, and to construct a second Gaussian blur kernel according to the second Gaussian blur kernel parameters.

[0160] The image multi-scale low-frequency component acquisition module 24 is used to process the original image based on the first Gaussian blur kernel to obtain the multi-scale low-frequency component of the original image; and process the enhanced image based on the second Gaussian blur kernel to obtain the multi-scale low-frequency component of the enhanced image.

[0161] The image multi-scale high-frequency component acquisition module 25 is used to extract the high-frequency components of the original image according to the multi-scale low-frequency components of the original image to obtain the multi-scale high-frequency components of the original image; and to extract the high-frequency components of the enhanced image according to the multi-scale low-frequency components of the enhanced image to obtain the multi-scale high-frequency components of the enhanced image.

[0162] The image multi-scale high-frequency component combining module 26 is used to combine the original image multi-scale high-frequency component and the enhanced image multi-scale high-frequency component to obtain multi-scale combined high-frequency component information.

[0163] The image high-frequency restoration module 27 is used to perform image restoration processing on the multi-scale combined high-frequency component information according to the high-frequency restoration technology to obtain a restored image.

[0164] Optionally, the Gaussian blur kernel parameter determination module 22 includes:

[0165] The image spectrum generating unit 221 is used to perform a two-dimensional discrete Fourier transform on the original image to obtain the original image spectrum; and perform a two-dimensional discrete Fourier transform on the enhanced image to obtain the enhanced image spectrum.

[0166] The parameter initial range confirmation unit 222 is used to determine the initial range of the standard deviation σ1 according to the distribution of the spectrum energy in the original image spectrum, and to determine the initial range of the standard deviation σ2 according to the distribution of the spectrum energy in the enhanced image spectrum.

[0167] The parameter confirmation unit 223 is used to iteratively search for the optimal σ1 value using the golden section search method within the initial range of the standard deviation σ1, and iteratively search for the optimal σ2 value using the golden section search method within the initial range of the standard deviation σ2.

[0168] Among them, the standard deviation σ1 of the first Gaussian blur kernel is the parameter of the first Gaussian blur kernel, and the standard deviation σ2 of the second Gaussian blur kernel is the parameter of the second Gaussian blur kernel.

[0169] Optionally, the parameter initial range confirmation unit 222 is specifically configured to perform the following steps:

[0170] S1: Denote the original image spectrum or the enhanced image spectrum as F(u, v), and calculate the spectral amplitude of F(u, v) Calculate the proportion P of the energy of F(u, v) in the low-frequency band to the total energy according to the following formula:

[0171]

[0172] Among them, Re[F(u, v)] is the real part of F(u, v), and Im[F(u, v)] is the imaginary part of F(u, v); the frequency range of the low-frequency band is (0, α×f max , α is a preset magnification factor, and f max is the maximum frequency of F(u, v), u, v ∈ L means that the values of u and v are within the low-frequency band, and u, v ∈ A means that the values of u and v are within the full frequency band of F(u, v);

[0173] S2: Determine the frequency interval [f1, f2] where the energy peak of F(u, v) is located, including:

[0174] Obtain the maximum spectral amplitude |F(u max , v max )| from the spectral amplitude |F(u, v)|; in F(u, v), with the maximum value point (u max , v max ) as the center, the sum of the spectral amplitudes within a circular region with a radius of r is S(r);

[0175] When holds, take the frequency range corresponding to [u max - r, v max - r] to [u max + r, v max + r] as the frequency interval [f1, f2]; where, Δr is a preset radius increment, and δ is a preset change rate threshold;

[0176] S3: Input the proportion P and into the trained neural network model, and output a1 and a2, where a1 < a2; if F(u, v) is the original image spectrum, the initial range of the standard deviation σ1 is [a1, a2], and if F(u, v) is the enhanced image spectrum, the initial range of the standard deviation σ2 is [a1, a2].

[0177] Optionally, the image low-frequency enhancement module 21 includes:

[0178] The feature extraction unit 211 is used to extract features from the original image using the trained low-frequency enhancement network to obtain local features and intermediate image features.

[0179] The feature aggregation unit 212 is used to aggregate the intermediate image features to obtain a feature aggregation result.

[0180] The global information extraction unit 213 is used to perform global feature extraction on the feature aggregation result to obtain global features.

[0181] The feature fusion unit 214 is used to fuse the local features and the global features using a feature fusion method based on the attention mechanism to obtain a fused feature.

[0182] The low-frequency enhanced image generating unit 215 is used to obtain a low-frequency enhanced image based on the fused features and the original image.

[0183] Optionally, the image low-frequency enhancement module 21 further includes a low-frequency enhancement network parameter setting unit 216, which is used to perform the following steps:

[0184] Use a sliding window to calculate the local variance of the original image;

[0185] Use the Sobel operator to calculate the horizontal and vertical gradients of the original image;

[0186] Based on the preset rules, the convolution kernel parameters of the low-frequency enhancement network are determined according to the local variance, horizontal gradient and vertical gradient.

[0187] Optionally, the image high frequency restoration module 27 includes:

[0188] The high-frequency component stacking unit 271 is used to stack high-frequency components of different scales in the multi-scale combined high-frequency component information along the channel dimension to form a comprehensive high-frequency component.

[0189] The Laplace component generating unit 272 is used to input the comprehensive high-frequency component into a preset high-frequency restoration model and output the Laplace component.

[0190] The low-frequency estimation image generation unit 273 is used to obtain a low-frequency estimation image by calculating the average value of each pixel in the original image and its neighboring pixels based on the mean filtering method according to the following formula:

[0191]

[0192] Where I(i+x,j+y) is the pixel value in the filter window of the original image, m and n are the number of rows and columns of the filter window respectively, and I lf(i, j) is the low-frequency estimation image; x and y are the coordinate variables relative to the center pixel in the filter window.

[0193] The image restoration unit 274 is used to add the Laplace component to the low-frequency estimation image pixel by pixel to obtain a restored image.

[0194] An embodiment of the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0195] An embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0196] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are only schematic, wherein the components described as separate parts may or may not be physically separated, and the parts displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0197] The above-mentioned embodiments only express several implementation methods of the embodiments of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent of the embodiments of the present application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the embodiments of the present application, and these all belong to the protection scope of the embodiments of the present application.

Claims

1. A multi-scale component based image restoration method, characterized in that: The method comprises: Perform low-frequency component enhancement processing on the original image to obtain an enhanced image; Determine a first Gaussian blur kernel parameter according to the original image, and determine a second Gaussian blur kernel parameter according to the enhanced image; Constructing a first Gaussian blur kernel according to the first Gaussian blur kernel parameters, and constructing a second Gaussian blur kernel according to the second Gaussian blur kernel parameters; Processing the original image based on the first Gaussian blur kernel to obtain a multi-scale low-frequency component of the original image; processing the enhanced image based on the second Gaussian blur kernel to obtain a multi-scale low-frequency component of the enhanced image; According to the multi-scale low-frequency components of the original image, high-frequency components of the original image are extracted to obtain multi-scale high-frequency components of the original image; according to the multi-scale low-frequency components of the enhanced image, high-frequency components of the enhanced image are extracted to obtain multi-scale high-frequency components of the enhanced image; Combining the multi-scale high-frequency component of the original image and the multi-scale high-frequency component of the enhanced image to obtain multi-scale combined high-frequency component information; According to the high-frequency restoration technology, image restoration processing is performed on the multi-scale combined high-frequency component information to obtain a restored image.

2. The method according to claim 1, characterized in that: The first Gaussian blur kernel parameter is the standard deviation σ1 of the first Gaussian blur kernel, and the second Gaussian blur kernel parameter is the standard deviation σ2 of the second Gaussian blur kernel; The determining of the first Gaussian blur kernel parameter according to the original image and the determining of the second Gaussian blur kernel parameter according to the enhanced image comprises: Performing a two-dimensional discrete Fourier transform on the original image to obtain an original image spectrum; performing a two-dimensional discrete Fourier transform on the enhanced image to obtain an enhanced image spectrum; Determine the initial range of the standard deviation σ1 according to the distribution of spectral energy in the original image spectrum, and determine the initial range of the standard deviation σ2 according to the distribution of spectral energy in the enhanced image spectrum; The golden section search method is used to iteratively search for the optimal σ1 value within the initial range of the standard deviation σ1, and the golden section search method is used to iteratively search for the optimal σ2 value within the initial range of the standard deviation σ2.

3. The method according to claim 2, characterized in that The step of determining the initial range of the standard deviation σ1 according to the distribution of spectral energy in the original image spectrum, and determining the initial range of the standard deviation σ2 according to the distribution of spectral energy in the enhanced image spectrum, comprises: S1: The original image spectrum or the enhanced image spectrum is recorded as F(u,v), and the spectrum amplitude of F(u,v) is calculated. The proportion P of the energy of F(u,v) in the low frequency band to the total energy is calculated according to the following formula: Where Re[F(u,v)] is the real part of F(u,v), Im[F(u,v)] is the imaginary part of F(u,v); the frequency range of the low frequency band is (0,α×f max ], α is the preset magnification, f max is the maximum frequency of F(u,v), u,v∈L means that the values ​​of u and v are within the low frequency band, and u,v∈A means that the values ​​of u and v are within the full frequency band of F(u,v); S2: Determine the frequency interval [f1,f2] where the energy peak of F(u,v) is located, including: From the spectrum amplitude |F(u,v)|, we can get the maximum spectrum amplitude |F(u max ,v max )|; In F(u,v), the maximum point (u max ,v max ) is the center and the sum of the spectrum amplitudes in the circular area with a radius of r is S(r); when When [u max -r,v max -r] to [u max +r,v max +r] as the frequency interval [f1, f2]; wherein Δr is a preset radius increase, and δ is a preset change rate threshold; S3: Input the ratio P and into the trained neural network model, and output a1 and a2, where a1 < a2; if F(u, v) is the spectrum of the original image, the initial range of the standard deviation σ1 is [a1, a2], and if F(u, v) is the spectrum of the enhanced image, the initial range of the standard deviation σ2 is [a1, a2].

4. The method according to claim 1, characterized in that: The step of performing low-frequency component enhancement processing on the original image to obtain an enhanced image includes: Using the trained low-frequency enhancement network to extract features from the original image, and obtain local features and intermediate image features; Aggregating the intermediate image features to obtain a feature aggregation result; Performing global feature extraction on the feature aggregation result to obtain global features; Using a feature fusion method based on an attention mechanism, the local features and the global features are fused to obtain fused features; The low-frequency enhanced image is obtained based on the fused features and the original image.

5. The method according to any one of claims 1 to 4, characterized in that: The feature fusion method based on the attention mechanism is used to fuse the local features and the global features to obtain fused features, including: Resizing the intermediate image features to obtain reshaped image features; Projecting the reshaped intermediate image features into a projection matrix to obtain projection features; Calculating a self-similarity matrix of the projection features; Using a neural network to learn the importance weights of the local features and the global features in different image regions based on the self-similarity matrix; The local features and the global features are fused according to the importance weights to obtain the fused features.

6. The method according to claim 5, characterized in that Before the original image is processed by the trained low-frequency enhancement network, the method further includes: Calculate the local variance of the original image using a sliding window; Using the Sobel operator to calculate the horizontal gradient and the vertical gradient of the original image; Based on a preset rule, the convolution kernel parameters of the low-frequency enhancement network are determined according to the local variance, the horizontal gradient, and the vertical gradient.

7. The method according to claim 6, characterized in that The method of performing image restoration processing on the multi-scale combined high-frequency component information according to the high-frequency restoration technology to obtain a restored image includes: Stacking high-frequency components of different scales in the multi-scale combined high-frequency component information along the channel dimension to form a comprehensive high-frequency component; Inputting the comprehensive high-frequency component into a preset high-frequency recovery model and outputting a Laplace component; According to the following formula, based on the mean filtering method, the low-frequency estimation image is obtained by calculating the average value of each pixel in the original image and its neighboring pixels: Where I(i+x,j+y) is the pixel value in the filter window of the original image, m and n are the number of rows and columns of the filter window respectively, and I lf (i, j) is the low-frequency estimation image; x and y are the coordinate variables relative to the center pixel in the filter window; The Laplace component is added to the low-frequency estimation image pixel by pixel to obtain the repaired image.

8. An image restoration device based on multi-scale components, characterized in that: The device comprises: An image low-frequency enhancement module is used to enhance the low-frequency components of the original image to obtain an enhanced image; A Gaussian blur kernel parameter determination module, used to determine a first Gaussian blur kernel parameter according to the original image, and to determine a second Gaussian blur kernel parameter according to the enhanced image; A Gaussian blur kernel construction module, configured to construct a first Gaussian blur kernel according to the first Gaussian blur kernel parameters, and to construct a second Gaussian blur kernel according to the second Gaussian blur kernel parameters; An image multi-scale low-frequency component acquisition module, used to process the original image based on the first Gaussian blur kernel to obtain the original image multi-scale low-frequency component; and process the enhanced image based on the second Gaussian blur kernel to obtain the enhanced image multi-scale low-frequency component; An image multi-scale high-frequency component acquisition module is used to extract high-frequency components from the original image according to the multi-scale low-frequency components of the original image to obtain multi-scale high-frequency components of the original image; and to extract high-frequency components from the enhanced image according to the multi-scale low-frequency components of the enhanced image to obtain multi-scale high-frequency components of the enhanced image; An image multi-scale high-frequency component combination module is used to combine the original image multi-scale high-frequency component and the enhanced image multi-scale high-frequency component to obtain multi-scale combined high-frequency component information; The image high-frequency restoration module is used to perform image restoration processing on the multi-scale combined high-frequency component information according to the high-frequency restoration technology to obtain a restored image.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images