An image restoration method based on restoration neural network model

Abstract

The present invention discloses an image restoration method based on a restoration neural network model, which belongs to the field of image processing technology. The present invention splits an RGB image into channel images, obtains the suspected blurred area of ​​each channel image respectively; extracts the neighborhood multi-directional image area according to the position of the suspected blurred area, and obtains the effective blurred area based on the spectrum difference screening; adjusts the spectrum of the blurred area according to the spectrum difference between the multi-directional image area and the effective blurred area to obtain two blurred adjustment areas; replaces the amplitude of the effective blurred area at the spectrum origin with the multi-directional image area to obtain two effective blurred areas with brightness adjustment; processes the two blurred adjustment areas and the two effective blurred areas with brightness adjustment through the restoration neural network model to obtain a restored area; finally, splices the restored areas of each channel to obtain an RGB restored image. This method improves the restoration accuracy of the blurred area of ​​the image through multi-dimensional spectrum analysis and neural network processing.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image restoration method based on a restoration neural network model. Background Art

[0002] During the actual imaging process, images often degrade due to various factors, such as the scattering of light by atmospheric particles resulting in reduced image contrast, color cast, and blurred details; insufficient light under low-light conditions resulting in high noise and loss of dark details; and camera shake or object movement causing image blur.

[0003] Existing technologies often use denoising convolutional neural networks (DnCNN) and super-resolution convolutional neural networks (SRCNN). These networks extract image features through multiple convolutional layers and learn the mapping relationship between degraded and original images in an end-to-end manner. However, the outlines of local areas are blurred, and their texture cannot be reflected. Therefore, when extracting features through neural networks, the lack of texture feature information leads to low restoration accuracy in blurred areas. Summary of the Invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides an image restoration method based on a restoration neural network model, which solves the problem of low restoration accuracy of fuzzy areas in the prior art.

[0005] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is: an image restoration method based on a restoration neural network model, comprising the following steps:

[0006] Split the RGB image into R, G, and B channel images;

[0007] Obtain the suspected blurred area for each channel image respectively;

[0008] In each channel image, according to the position of the suspected fuzzy area, the image area of ​​multiple directions in the neighborhood is extracted. Based on the difference in spectrum between the image area and the suspected fuzzy area, the suspected fuzzy area is screened to obtain the effective fuzzy area.

[0009] According to the difference in spectrum between the image area in different directions and the effective blur area, the spectrum of the blur area is adjusted to obtain two blur adjustment areas;

[0010] The amplitude of the effective fuzzy area at the origin of the spectrum is replaced by image areas of different orientations to obtain two effective fuzzy areas with brightness adjustment.

[0011] The two fuzzy adjustment areas and the two effective fuzzy areas of brightness adjustment are processed using a restoration neural network model to obtain the restoration area.

[0012] The restored areas corresponding to the R, G, and B channels are spliced ​​to obtain the RGB restored image.

[0013] Furthermore, the process of obtaining the suspected blur area includes:

[0014] On each channel image, calculate the difference between the pixel value of each central pixel and the value of each pixel in the neighborhood to obtain the pixel difference;

[0015] Filter out pixel differences greater than 0, and average the pixel differences greater than 0 to obtain the pixel threshold;

[0016] All pixel differences in the neighborhood are lower than the pixel threshold, and at least one central pixel in the neighborhood has a non-zero pixel difference, which is marked as a fuzzy point;

[0017] The horizontal width and vertical height of the connected area formed by each blurred point are extracted, and the area of ​​horizontal width × vertical height is cut out from the image as the suspected blurred area, and the connected area is located within the suspected blurred area.

[0018] Furthermore, the process of obtaining the effective fuzzy area includes:

[0019] Performing Fourier transform on the image area and the suspected blurred area to obtain the frequency spectrum of the image area and the suspected blurred area;

[0020] Add the corresponding amplitudes of the high-frequency components in the spectrum of the image area to obtain the high-frequency amplitude energy of the neighborhood;

[0021] Add the corresponding amplitudes of the low-frequency components in the spectrum of the image area to obtain the low-frequency amplitude energy of the neighborhood;

[0022] Add the corresponding amplitudes of the high-frequency components in the spectrum of the suspected fuzzy area to obtain the suspected high-frequency amplitude energy;

[0023] Add the corresponding amplitudes of the low-frequency components in the spectrum of the suspected fuzzy area to obtain the suspected low-frequency amplitude energy;

[0024] Subtract the suspected high-frequency amplitude energy from the mean of the neighborhood high-frequency amplitude energy of all image regions, and take the absolute value to obtain the high-frequency amplitude energy difference;

[0025] Subtract the suspected low-frequency amplitude energy from the mean of the neighborhood low-frequency amplitude energy of all image regions, and take the absolute value to obtain the low-frequency amplitude energy difference;

[0026] When the high-frequency amplitude energy difference and the low-frequency amplitude energy difference are both greater than the amplitude energy difference threshold, the corresponding suspected fuzzy area is a valid fuzzy area.

[0027] Furthermore, the process of obtaining two fuzzy adjustment regions includes:

[0028] According to the frequency spectrum differences between the image areas in the four directions of up, down, left and right and the effective fuzzy area, the frequency spectrum of the fuzzy area is adjusted to obtain a first fuzzy adjustment area;

[0029] According to the frequency spectrum differences between the image areas at the upper left, upper right, lower left and lower right and the effective fuzzy area, the frequency spectrum of the fuzzy area is adjusted to obtain a second fuzzy adjustment area.

[0030] Furthermore, the process of obtaining the first fuzzy adjustment region and the process of obtaining the second fuzzy adjustment region both include:

[0031] Calculate the amplitude mean of the high-frequency components of the image area spectrum in four directions to obtain the neighborhood high-frequency amplitude mean;

[0032] Calculate the amplitude mean of the low-frequency component of the spectrum of the image area in four directions to obtain the neighborhood low-frequency amplitude mean;

[0033] Calculate the amplitude mean of the high-frequency components in the spectrum of the fuzzy area to obtain the fuzzy high-frequency amplitude mean, and calculate the amplitude mean of the low-frequency components in the spectrum of the fuzzy area to obtain the fuzzy low-frequency amplitude mean;

[0034] The high-frequency amplitude difference is obtained by subtracting the neighborhood high-frequency amplitude mean from the fuzzy high-frequency amplitude mean, and the low-frequency amplitude difference is obtained by subtracting the neighborhood low-frequency amplitude mean from the fuzzy low-frequency amplitude mean;

[0035] The high-frequency component of the fuzzy region spectrum is subtracted from the high-frequency amplitude difference from each amplitude to obtain the adjusted high-frequency component, and the low-frequency component of the fuzzy region spectrum is subtracted from the low-frequency amplitude difference from each amplitude to obtain the adjusted low-frequency component;

[0036] Perform inverse Fourier transform on the adjusted high-frequency component and the adjusted low-frequency component to obtain a fuzzy adjustment area.

[0037] Furthermore, the process of obtaining two effective blurred areas for brightness adjustment includes:

[0038] Obtain the average of the spectrum origin amplitudes of the four azimuth image regions of upper, lower, left, and right to obtain the first origin amplitude mean;

[0039] Obtain the average of the spectrum origin amplitudes of the four azimuth image regions at upper left, upper right, lower left, and lower right to obtain the second origin amplitude mean;

[0040] Replacing the amplitude at the origin of the spectrum of the effective fuzzy area with the mean amplitude of the first origin, and performing inverse Fourier transform on the replaced spectrum to obtain the effective fuzzy area for the first brightness adjustment;

[0041] The amplitude at the origin of the spectrum of the effective fuzzy area is replaced by the second origin amplitude mean, and the replaced spectrum is subjected to inverse Fourier transform to obtain the effective fuzzy area for the second brightness adjustment.

[0042] Furthermore, the restoration neural network model includes: a first fuzzy region enhancement module, a second fuzzy region enhancement module, a first restoration module, a second restoration module and an adder A1;

[0043] The first input end of the first fuzzy area enhancement module is used to input the first fuzzy adjustment area, the second input end thereof is used to input the first effective fuzzy area for brightness adjustment, and the output end thereof is connected to the input end of the first restoration module;

[0044] The first input end of the second fuzzy area enhancement module is used to input the second fuzzy adjustment area, the second input end thereof is used to input the effective fuzzy area of ​​the second brightness adjustment, and the output end thereof is connected to the input end of the second restoration module;

[0045] The input end of the adder A1 is connected to the output end of the first restoration module and the output end of the second restoration module respectively, and the output end of the adder A1 serves as the output end of the restoration neural network model.

[0046] Furthermore, the expression of the first fuzzy area enhancement module is: , where I z,1 is the output of the first fuzzy area enhancement module, I adj,1 is the first fuzzy adjustment area, I E,1 is the effective blur area for the first brightness adjustment, is a 1×1 convolutional layer. is element-wise addition;

[0047] The expression of the second fuzzy area enhancement module is: , where I z,2 is the output of the second fuzzy area enhancement module, I adj,2 is the second fuzzy adjustment area, I E,2 The effective blur area for the second brightness adjustment, It is a convolutional layer of size 1×1.

[0048] Furthermore, the first restoration module and the second restoration module both include: a first convolution layer, a second convolution layer, a downsampling layer, a third convolution layer, a first upsampling layer, a fourth convolution layer, a second upsampling layer, a fifth convolution layer and a sixth convolution layer connected in sequence.

[0049] The beneficial effects of the present invention are:

[0050] The present invention splits the RGB image into three-channel images, obtains the suspected blurred area for each channel image, and then extracts the multi-directional neighborhood image area of ​​the suspected blurred area (such as up, down, left, right, and diagonal directions) in each channel. By comparing the frequency spectrum gap between the neighborhood and the blurred area, the interference of the non-blurred area is effectively eliminated, and the real and effective blurred area is screened out.

[0051] According to the spectrum gap between different azimuth neighborhoods and the effective fuzzy area, the spectrum of the fuzzy area is adjusted in a targeted manner to generate two fuzzy adjustment areas to highlight the contour details.

[0052] Then, by replacing the amplitude of the effective blurred area at the origin of the spectrum, two effective blurred areas with brightness adjustment are generated, and the overall brightness distribution of the blurred area is accurately controlled.

[0053] The present invention simultaneously inputs two fuzzy adjustment areas (rich in edge detail information) and two effective fuzzy areas for brightness adjustment into the restoration neural network model, providing the restoration neural network model with richer input features, enabling the model to simultaneously learn the edge structure and brightness distribution rules, solving the problem of low restoration accuracy of fuzzy areas due to the lack of texture feature information in the existing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a flowchart of an image restoration method based on a restoration neural network model.

[0055] Figure 2 It is the position relationship diagram of the connected area and the suspected fuzzy area;

[0056] Figure 3 is a schematic diagram of a blurred area image;

[0057] Figure 4 Schematic diagram of the blurred area and the upper, lower, left, right, upper left, upper right, lower left, and lower right image areas;

[0058] Figure 5 Schematic diagram of the structure of the restored neural network model;

[0059] Figure 6 It is a structural diagram of the first restoration module and the second restoration module. DETAILED DESCRIPTION

[0060] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0061] like Figure 1 As shown, an image restoration method based on a restoration neural network model includes the following steps:

[0062] Split the RGB image into R, G, and B channel images;

[0063] Obtain the suspected blurred area for each channel image respectively;

[0064] In each channel image, according to the position of the suspected fuzzy area, the image area of ​​multiple directions in the neighborhood is extracted. Based on the difference in spectrum between the image area and the suspected fuzzy area, the suspected fuzzy area is screened to obtain the effective fuzzy area.

[0065] According to the difference in spectrum between the image area in different directions and the effective blur area, the spectrum of the blur area is adjusted to obtain two blur adjustment areas;

[0066] The amplitude of the effective fuzzy area at the origin of the spectrum is replaced by image areas of different orientations to obtain two effective fuzzy areas with brightness adjustment.

[0067] The two fuzzy adjustment areas and the two effective fuzzy areas of brightness adjustment are processed using a restoration neural network model to obtain the restoration area.

[0068] The restored areas corresponding to the R, G, and B channels are spliced ​​to obtain the RGB restored image.

[0069] In this embodiment, the process of obtaining the suspected blur area includes:

[0070] On each channel image, calculate the difference between the pixel value of each central pixel and each pixel value in the neighborhood to obtain the pixel difference, where the pixel value is the channel value of the channel image, and the pixel difference is the absolute value of the difference between the two pixel values;

[0071] Filter out pixel differences greater than 0, and average the pixel differences greater than 0 to obtain the pixel threshold;

[0072] All pixel differences in the neighborhood are lower than the pixel threshold, and at least one central pixel in the neighborhood has a non-zero pixel difference, which is marked as a fuzzy point;

[0073] The horizontal width and vertical height of the connected area formed by each blurred point are extracted, and the area of ​​horizontal width × vertical height is cut out from the image as the suspected blurred area, and the connected area is located within the suspected blurred area.

[0074] The horizontal width = the maximum horizontal coordinate of the right boundary of the connected area minus the minimum horizontal coordinate of the left boundary, and the vertical height = the maximum vertical coordinate of the upper boundary minus the minimum vertical coordinate of the lower boundary, such as Figure 2 shown.

[0075] This method is applicable to blurred areas with gradual pixel value changes, unclear boundaries, and smooth pixel value transitions. In practical applications, some image degradation scenarios (such as distant blur caused by atmospheric scattering and diffuse blur caused by out-of-focus imaging) lack a clear grayscale abrupt boundary between blurred and normal areas, and pixel values ​​exhibit a gradual change. For such scenarios, this method uses a dynamic threshold determination mechanism based on the difference between pixel neighborhoods to identify pixels with low pixel disparity and non-zero pixel disparity as blurred points. When the pixel disparity is too large, the boundary at that pixel is clear and it is not a blurred point.

[0076] The present invention is applicable to the situation where the pixel value in the fuzzy area changes gradually, such as Figure 3 As shown, the boundary of the image is not clear, the pixel values ​​are slowly transitioned, and there is no clear boundary.

[0077] In this embodiment, the process of obtaining the effective fuzzy area includes:

[0078] Performing Fourier transform on the image area and the suspected blurred area to obtain the frequency spectrum of the image area and the suspected blurred area;

[0079] Add the corresponding amplitudes of the high-frequency components in the spectrum of the image area to obtain the high-frequency amplitude energy of the neighborhood;

[0080] Add the corresponding amplitudes of the low-frequency components in the spectrum of the image area to obtain the low-frequency amplitude energy of the neighborhood;

[0081] Add the corresponding amplitudes of the high-frequency components in the spectrum of the suspected fuzzy area to obtain the suspected high-frequency amplitude energy;

[0082] Add the corresponding amplitudes of the low-frequency components in the spectrum of the suspected fuzzy area to obtain the suspected low-frequency amplitude energy;

[0083] Subtract the suspected high-frequency amplitude energy from the mean of the neighborhood high-frequency amplitude energy of all image regions, and take the absolute value to obtain the high-frequency amplitude energy difference;

[0084] Subtract the suspected low-frequency amplitude energy from the mean of the neighborhood low-frequency amplitude energy of all image regions, and take the absolute value to obtain the low-frequency amplitude energy difference;

[0085] When the high-frequency amplitude energy difference and the low-frequency amplitude energy difference are both greater than the amplitude energy difference threshold, the corresponding suspected fuzzy area is a valid fuzzy area.

[0086] The present invention performs Fourier transform on the image area and the suspected blurred area respectively to obtain high-frequency components and low-frequency components. By comparing the high-frequency and low-frequency amplitude energy differences between the suspected blurred area and the neighboring image area and combining threshold judgment, accurate screening of the blurred area is achieved to prevent normal areas from being misjudged as blurred areas.

[0087] In this embodiment, the amplitude energy difference threshold can be set through experiments or experience.

[0088] In this embodiment, the process of obtaining two fuzzy adjustment areas includes:

[0089] According to the frequency spectrum differences between the image areas in the four directions of up, down, left and right and the effective fuzzy area, the frequency spectrum of the fuzzy area is adjusted to obtain a first fuzzy adjustment area;

[0090] According to the frequency spectrum differences between the image areas at the upper left, upper right, lower left and lower right and the effective fuzzy area, the frequency spectrum of the fuzzy area is adjusted to obtain a second fuzzy adjustment area.

[0091] like Figure 4 As shown, the positional relationship between the blurred area and the image areas above, below, left, right, upper left, upper right, lower left, and lower right.

[0092] In this embodiment, the process of obtaining the first fuzzy adjustment area and the process of obtaining the second fuzzy adjustment area both include:

[0093] Calculate the amplitude mean of the high-frequency components of the image area spectrum in the four directions to obtain the neighborhood high-frequency amplitude mean, which is the sum of all amplitudes of the high frequencies in the four directions / the number of frequency points of the high-frequency components in the four directions;

[0094] Calculate the amplitude mean of the low-frequency components of the image area spectrum in the four directions to obtain the neighborhood low-frequency amplitude mean, which is the sum of all amplitudes of the four directions low-frequency components / the number of frequency points of the four directions low-frequency components;

[0095] Calculate the amplitude mean of the high-frequency components in the spectrum of the blurred area (here, the amplitude mean = all amplitudes of the high-frequency components in the blurred area and / or the number of frequency points of the high-frequency components in the blurred area) to obtain the blurred high-frequency amplitude mean; calculate the amplitude mean of the low-frequency components in the spectrum of the blurred area (here, the amplitude mean = all amplitudes of the low-frequency components in the blurred area and / or the number of frequency points of the low-frequency components in the blurred area) to obtain the blurred low-frequency amplitude mean;

[0096] The high-frequency amplitude difference is obtained by subtracting the neighborhood high-frequency amplitude mean from the fuzzy high-frequency amplitude mean, and the low-frequency amplitude difference is obtained by subtracting the neighborhood low-frequency amplitude mean from the fuzzy low-frequency amplitude mean;

[0097] The high-frequency component of the fuzzy region spectrum is subtracted from the high-frequency amplitude difference from each amplitude to obtain the adjusted high-frequency component, and the low-frequency component of the fuzzy region spectrum is subtracted from the low-frequency amplitude difference from each amplitude to obtain the adjusted low-frequency component;

[0098] Perform inverse Fourier transform on the adjusted high-frequency component and the adjusted low-frequency component to obtain a fuzzy adjustment area.

[0099] In the frequency domain, high-frequency components of an image primarily correspond to image details, edges, and texture information, while low-frequency components reflect the overall image information. In this embodiment, by calculating the mean high- and low-frequency amplitudes of the spectra of the four azimuth image regions and the blurred region, and comparing the difference to obtain the amplitude difference, the spectral components of the blurred region are adjusted based on the amplitude difference, thereby achieving refined correction of both the details and the overall structure of the blurred region.

[0100] In actual scenes, due to defects in imaging equipment, interference from the shooting environment and other factors, images are prone to blurring, and there are obvious differences in spectral characteristics between the blurred area and the surrounding normal area. This method uses this spectral difference to adjust the blurred area in a targeted manner. The first blurred adjustment area obtained can adjust the details and structure of the blurred area in the horizontal and vertical directions through spectral comparison of the four directional image areas of upper, lower, left and right, so that the blurred area can be better integrated with the surrounding area in the horizontal and vertical directions; the second blurred adjustment area uses the spectral comparison of the four directional image areas of upper left, upper right, lower left and lower right to further optimize the blurred area from an oblique angle and improve its details and overall shape. In this way, the present invention finally obtains a blurred adjustment area that is closer to the surrounding area in spectral characteristics.

[0101] In this embodiment, the process of obtaining two effective blurred areas for brightness adjustment includes:

[0102] Obtain the average of the spectrum origin amplitudes of the four azimuth image regions of upper, lower, left, and right to obtain the first origin amplitude mean;

[0103] Obtain the average of the spectrum origin amplitudes of the four azimuth image regions at upper left, upper right, lower left, and lower right to obtain the second origin amplitude mean;

[0104] The amplitude at the origin of the spectrum of the effective fuzzy area is replaced by the amplitude mean of the first origin (the amplitude at other positions remains unchanged), and the replaced spectrum is subjected to inverse Fourier transform to obtain the effective fuzzy area of ​​the first brightness adjustment;

[0105] The amplitude at the origin of the spectrum of the effective fuzzy area is replaced by the second origin amplitude mean (the amplitudes at other positions remain unchanged), and the replaced spectrum is subjected to inverse Fourier transform to obtain the effective fuzzy area for the second brightness adjustment.

[0106] In the frequency domain representation of an image, the amplitude of the spectrum origin is closely related to the image's average brightness. From the perspective of signal processing, after the Fourier transform converts the image from the spatial domain to the frequency domain, the spectrum origin corresponds to the image's DC component, which essentially reflects the image's overall brightness. Therefore, adjusting the amplitude of the spectrum origin can directly change the image's average brightness.

[0107] In actual imaging, due to uneven lighting and improper camera parameter settings, different image regions may experience brightness differences. This is particularly true for blurred areas, where the brightness often differs from that of the surrounding normal areas, affecting the overall visual quality of the image. This method calculates the average amplitude of the spectrum origins for the image regions in the four directions (up, down, left, and right), as well as the upper left, upper right, lower left, and lower right. These two sets of means are actually statistical results based on the brightness information of the neighborhood in different directions, representing the average brightness level of the surrounding normal areas.

[0108] These two sets of means are used to replace the amplitude at the origin of the spectrum of the effective blurred area, and an inverse Fourier transform is performed, which is equivalent to adjusting the brightness of the blurred area to a level that matches the surrounding area. The effective blurred area of ​​the first brightness adjustment is adjusted based on the brightness mean of the horizontal and vertical neighborhoods, which can better blend the brightness of the blurred area with the surrounding area in the horizontal and vertical directions; the effective blurred area of ​​the second brightness adjustment is adjusted according to the brightness mean of the diagonal neighborhood, further improving the brightness consistency of the blurred area from multiple angles. In this way, the brightness imbalance problem between the blurred area and the surrounding area is effectively improved, and the uniformity of the overall brightness of the image is improved.

[0109] like Figure 5 As shown, the restoration neural network model includes: a first fuzzy region enhancement module, a second fuzzy region enhancement module, a first restoration module, a second restoration module and an adder A1;

[0110] The first input end of the first fuzzy area enhancement module is used to input the first fuzzy adjustment area, the second input end thereof is used to input the first effective fuzzy area for brightness adjustment, and the output end thereof is connected to the input end of the first restoration module;

[0111] The first input end of the second fuzzy area enhancement module is used to input the second fuzzy adjustment area, the second input end thereof is used to input the effective fuzzy area of ​​the second brightness adjustment, and the output end thereof is connected to the input end of the second restoration module;

[0112] The input end of the adder A1 is connected to the output end of the first restoration module and the output end of the second restoration module respectively, and the output end of the adder A1 serves as the output end of the restoration neural network model.

[0113] In this embodiment, the expression of the first fuzzy area enhancement module is: , where I z,1 is the output of the first fuzzy area enhancement module, I adj,1 is the first fuzzy adjustment area, I E,1 is the effective blur area for the first brightness adjustment, is a 1×1 convolutional layer. is element-wise addition;

[0114] The expression of the second fuzzy area enhancement module is: , where I z,2 is the output of the second fuzzy area enhancement module, I adj,2 is the second fuzzy adjustment area, I E,2 The effective blur area for the second brightness adjustment, It is a convolutional layer of size 1×1.

[0115] The same type of orientation-optimized blurred adjustment area and the effective blurred area with brightness adjustment are input into the same blurred area enhancement module. The features of each channel value in the area are extracted through a 1×1 convolution layer, and then added element by element. The blurred area features are enhanced from the dual dimensions of "detail restoration + brightness coordination", so that the texture detail features are integrated with the overall channel feature values.

[0116] The present invention restores features of one type of orientation through a first restoration module, restores features of another type of orientation through a second restoration module, and then adds them together through adder A1 to fuse the restored image features of horizontal / vertical orientation and oblique orientation to improve the restoration accuracy.

[0117] like Figure 6 As shown, the first restoration module and the second restoration module both include: a first convolution layer, a second convolution layer, a downsampling layer, a third convolution layer, a first upsampling layer, a fourth convolution layer, a second upsampling layer, a fifth convolution layer and a sixth convolution layer connected in sequence.

[0118] The present invention extracts features of different levels of the image from shallow to deep through multiple groups of convolutional layers. The downsampling layer compresses the size and captures the macro structure to avoid repairing and ignoring global coordination. The upsampling layer restores the size and integrates the details to fill in the lost information.

[0119] The present invention splits the RGB image into three-channel images, obtains the suspected blurred area for each channel image, and then extracts the multi-directional neighborhood image area of ​​the suspected blurred area (such as up, down, left, right, and diagonal directions) in each channel. By comparing the frequency spectrum gap between the neighborhood and the blurred area, the interference of the non-blurred area is effectively eliminated, and the real and effective blurred area is screened out.

[0120] According to the spectrum gap between different azimuth neighborhoods and the effective fuzzy area, the spectrum of the fuzzy area is adjusted in a targeted manner to generate two fuzzy adjustment areas to highlight the contour details.

[0121] Then, by replacing the amplitude of the effective blurred area at the origin of the spectrum, two effective blurred areas with brightness adjustment are generated, and the overall brightness distribution of the blurred area is accurately controlled.

[0122] The present invention simultaneously inputs two fuzzy adjustment areas (rich in edge detail information) and two effective fuzzy areas for brightness adjustment into the restoration neural network model, providing the restoration neural network model with richer input features, enabling the model to simultaneously learn the edge structure and brightness distribution rules, solving the problem of low restoration accuracy of fuzzy areas due to the lack of texture feature information in the existing technology.

[0123] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. An image restoration method based on a restoration neural network model, characterized in that: The following steps are involved: Split the RGB image into R, G, and B channel images; Obtain the suspected blurred area for each channel image respectively; In each channel image, according to the position of the suspected fuzzy area, the image area of ​​multiple directions in the neighborhood is extracted. Based on the difference in spectrum between the image area and the suspected fuzzy area, the suspected fuzzy area is screened to obtain the effective fuzzy area. According to the difference in spectrum between the image area in different directions and the effective blur area, the spectrum of the blur area is adjusted to obtain two blur adjustment areas: Calculate the amplitude mean of the high-frequency components of the image area spectrum in four directions to obtain the neighborhood high-frequency amplitude mean; Calculate the amplitude mean of the low-frequency component of the spectrum of the image area in four directions to obtain the neighborhood low-frequency amplitude mean; Calculate the amplitude mean of the high-frequency components in the spectrum of the fuzzy area to obtain the fuzzy high-frequency amplitude mean, and calculate the amplitude mean of the low-frequency components in the spectrum of the fuzzy area to obtain the fuzzy low-frequency amplitude mean; The high-frequency amplitude difference is obtained by subtracting the neighborhood high-frequency amplitude mean from the fuzzy high-frequency amplitude mean, and the low-frequency amplitude difference is obtained by subtracting the neighborhood low-frequency amplitude mean from the fuzzy low-frequency amplitude mean; The high-frequency component of the fuzzy region spectrum is subtracted from the high-frequency amplitude difference from each amplitude to obtain the adjusted high-frequency component, and the low-frequency component of the fuzzy region spectrum is subtracted from the low-frequency amplitude difference from each amplitude to obtain the adjusted low-frequency component; Performing inverse Fourier transform on the adjusted high-frequency component and the adjusted low-frequency component to obtain a fuzzy adjustment area; The amplitude of the effective fuzzy area at the origin of the spectrum is replaced by image areas of different orientations to obtain two effective fuzzy areas with brightness adjustment. The two fuzzy adjustment areas and the two effective fuzzy areas of brightness adjustment are processed using a restoration neural network model to obtain the restoration area. The restored areas corresponding to the R, G, and B channels are spliced ​​to obtain the RGB restored image.

2. The image restoration method based on the restoration neural network model according to claim 1, characterized in that: The process of obtaining the suspected fuzzy area includes: On each channel image, calculate the difference between the pixel value of each central pixel and the value of each pixel in the neighborhood to obtain the pixel difference; Filter out pixel differences greater than 0, and average the pixel differences greater than 0 to obtain the pixel threshold; For a central pixel, if the difference between all pixels in the neighborhood is lower than the pixel threshold and there is at least one pixel difference in the neighborhood that is not 0, the central pixel is marked as a fuzzy point. The horizontal width and vertical height of the connected area formed by each blurred point are extracted, and the area of ​​horizontal width × vertical height is cut out from the image as the suspected blurred area, and the connected area is located within the suspected blurred area.

3. The image restoration method based on the restoration neural network model according to claim 1, characterized in that: The process of obtaining the effective fuzzy area includes: Performing Fourier transform on the image area and the suspected blurred area to obtain the frequency spectrum of the image area and the suspected blurred area; Add the corresponding amplitudes of the high-frequency components in the spectrum of the image area to obtain the high-frequency amplitude energy of the neighborhood; Add the corresponding amplitudes of the low-frequency components in the spectrum of the image area to obtain the low-frequency amplitude energy of the neighborhood; Add the corresponding amplitudes of the high-frequency components in the spectrum of the suspected fuzzy area to obtain the suspected high-frequency amplitude energy; Add the corresponding amplitudes of the low-frequency components in the spectrum of the suspected fuzzy area to obtain the suspected low-frequency amplitude energy; Subtract the suspected high-frequency amplitude energy from the mean of the neighborhood high-frequency amplitude energy of all image regions, and take the absolute value to obtain the high-frequency amplitude energy difference; Subtract the suspected low-frequency amplitude energy from the mean of the neighborhood low-frequency amplitude energy of all image regions, and take the absolute value to obtain the low-frequency amplitude energy difference; When the high-frequency amplitude energy difference and the low-frequency amplitude energy difference are both greater than the amplitude energy difference threshold, the corresponding suspected fuzzy area is a valid fuzzy area.

4. The image restoration method based on the restoration neural network model according to claim 1, characterized in that: The process of obtaining two fuzzy adjustment areas includes: According to the frequency spectrum differences between the image areas in the four directions of up, down, left and right and the effective fuzzy area, the frequency spectrum of the fuzzy area is adjusted to obtain a first fuzzy adjustment area; According to the frequency spectrum differences between the image areas at the upper left, upper right, lower left and lower right and the effective fuzzy area, the frequency spectrum of the fuzzy area is adjusted to obtain a second fuzzy adjustment area.

5. The image restoration method based on the restoration neural network model according to claim 1, characterized in that: The process of obtaining two effective fuzzy areas for brightness adjustment includes: Obtain the average of the spectrum origin amplitudes of the four azimuth image regions of upper, lower, left, and right to obtain the first origin amplitude mean; Obtain the average of the spectrum origin amplitudes of the four azimuth image regions at upper left, upper right, lower left, and lower right to obtain the second origin amplitude mean; Replacing the amplitude at the origin of the spectrum of the effective fuzzy area with the mean amplitude of the first origin, and performing inverse Fourier transform on the replaced spectrum to obtain the effective fuzzy area for the first brightness adjustment; The amplitude at the origin of the spectrum of the effective fuzzy area is replaced by the second origin amplitude mean, and the replaced spectrum is subjected to inverse Fourier transform to obtain the effective fuzzy area for the second brightness adjustment.

6. The image restoration method based on the restoration neural network model according to claim 1, characterized in that: The restoration neural network model includes: a first fuzzy region enhancement module, a second fuzzy region enhancement module, a first restoration module, a second restoration module and an adder A1; The first input end of the first fuzzy area enhancement module is used to input the first fuzzy adjustment area, the second input end thereof is used to input the first effective fuzzy area for brightness adjustment, and the output end thereof is connected to the input end of the first restoration module; The first input end of the second fuzzy area enhancement module is used to input the second fuzzy adjustment area, the second input end thereof is used to input the effective fuzzy area of ​​the second brightness adjustment, and the output end thereof is connected to the input end of the second restoration module; The input end of the adder A1 is connected to the output end of the first restoration module and the output end of the second restoration module respectively, and the output end of the adder A1 serves as the output end of the restoration neural network model.

7. The image restoration method based on the restoration neural network model according to claim 6, characterized in that: The expression of the first fuzzy area enhancement module is: , where I z,1 is the output of the first fuzzy area enhancement module, I adj,1 is the first fuzzy adjustment area, I E,1 is the effective blur area for the first brightness adjustment, is a 1×1 convolutional layer. is element-wise addition; The expression of the second fuzzy area enhancement module is: , where I z,2 is the output of the second fuzzy area enhancement module, I adj,2 is the second fuzzy adjustment area, I E,2 The effective blur area for the second brightness adjustment.

8. The image restoration method based on the restoration neural network model according to claim 6, characterized in that: The first restoration module and the second restoration module both include: a first convolution layer, a second convolution layer, a downsampling layer, a third convolution layer, a first upsampling layer, a fourth convolution layer, a second upsampling layer, a fifth convolution layer and a sixth convolution layer connected in sequence.

Images