Super-resolution method

Abstract

The invention provides a super-resolution method, which is a super-resolution method based on combination of a Gaussian model and deep learning. Compared with a traditional mode, the method has the advantages that on the basis of slightly increasing power consumption, the effect far better than the performance of the method is achieved, and the obtained high-resolution image is clearer. The circuit structure does not need to be supported by special computing units (NPU, TPU and the like), and high performance is achieved at low cost. The SSIM (Structural Similarity Index) index and the PSNR (Peak Signal to Noise Ratio) index of the high-resolution image obtained by adopting the super-resolution method reach the advanced level; and on the basis that the image index reaches the advanced level and the image is clear, the method disclosed by the invention greatly reduces the required memory.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and specifically relates to a super-resolution method. Background Technology

[0002] Super-resolution (URP) is a process of increasing the resolution of an original image using hardware or software methods. It involves creating a high-resolution image from a series of low-resolution images; URP converts low-resolution images or videos into high-resolution ones. URP has significant applications in many fields, including security monitoring, medical imaging, mobile photography, and video display.

[0003] There are two main ways to achieve super-resolution: traditional and complex neural network methods. Traditional methods have advantages such as simple logic, ease of implementation, and low power consumption, but the restored image is generally blurry. Products using this method primarily employ it for magnification. Complex neural network methods restore images that are clearer, sharper, and more natural, but they are logically complex, consume more power, and require specialized computing units such as NPUs (Neural Processing Units). Summary of the Invention

[0004] The purpose of this invention is to provide a super-resolution method that, in terms of circuit structure, requires no special computing units (NPU, TPU, etc.) to support it, achieving high performance at a low cost. The high-resolution images obtained using the super-resolution method of this invention achieve state-of-the-art SSIM (Structural Similarity Index) and PSNR (Peak Signal-to-Noise Ratio) metrics; furthermore, while achieving state-of-the-art image metrics and image clarity, the method of this invention significantly reduces the required memory. Super-resolution technology is achieved in a low-cost manner.

[0005] This invention provides a super-resolution method, comprising:

[0006] S1. Decompose the low-resolution image of size h*w into a low-bit image of size h*w and a high-bit image of size h*w using a high-low bit decomposition operation;

[0007] S2. The high-bit image and the low-bit image are processed by the first sub-neural network to extract and enhance features, resulting in a first feature image with edge enhancement of size h*w.

[0008] S3. After the first feature image is processed by the second sub-neural network, a second feature image of size h*w is obtained, which fully reflects the high-frequency and low-frequency information of the image.

[0009] S4. The first feature image, the second feature image, and the target image are interpolated using a two-dimensional Gaussian distribution as the interpolation kernel to reconstruct a high-resolution image of size H*W; the target image is the blank image before the high-resolution image is assigned pixel values.

[0010] Furthermore, in step S2, the first sub-neural network includes a first high-bit sub-neural network and a first low-bit sub-neural network;

[0011] The first high-bit sub-neural network extracts and enhances the high-frequency edge information contained in the high-bit image to obtain a high-bit edge enhanced image; the first low-bit sub-neural network extracts and enhances the high-frequency edge information contained in the low-bit image to obtain a low-bit edge enhanced image.

[0012] The first feature image is obtained by adding the pixel values ​​at all the same pixel positions in the high-bit edge enhancement image and the low-bit edge enhancement image.

[0013] Furthermore, for the high-bit-level image and the low-bit-level image, prior knowledge from known image processing is incorporated into the first sub-neural network. Through convolution operations, the first sub-neural network model can better distinguish the features of edges at different angles in the image, thereby completing the extraction and enhancement of high-frequency edge features.

[0014] Furthermore, for the high-bit image, the first high-bit sub-neural network employs three different shapes of high-bit convolution kernels; the first type of high-bit convolution kernel takes three adjacent pixels located in the same row; the second type of high-bit convolution kernel takes three adjacent pixels located on the diagonal; and the third type of high-bit convolution kernel takes three adjacent pixels located in different rows and columns; the receptive field of each type of high-bit convolution kernel is 1*3.

[0015] Furthermore, the high-bit images rotated by 0 degrees, 90 degrees, 180 degrees, and 270 degrees are subjected to convolution operations primarily using the first type of high-bit convolution kernel to generate corresponding outputs; the high-bit images rotated by 0 degrees, 90 degrees, 180 degrees, and 270 degrees are subjected to convolution operations primarily using the second type of high-bit convolution kernel to generate corresponding outputs; and the high-bit images rotated by 0 degrees, 90 degrees, 180 degrees, and 270 degrees are subjected to convolution operations primarily using the third type of high-bit convolution kernel to generate corresponding outputs.

[0016] The three outputs described above are rotated in the opposite direction by 0 degrees, 90 degrees, 180 degrees and 270 degrees respectively. The average of the twelve results after the reverse rotation is taken as the first average, which is the high-bit edge enhancement image.

[0017] Furthermore, for the low-bit image, the first low-bit sub-neural network employs two different shapes of low-bit convolution kernels; the first low-bit convolution kernel takes two adjacent pixels located in the same row; the second low-bit convolution kernel takes two adjacent pixels located on the diagonal; the receptive field of each low-bit convolution kernel is 1*2; high-frequency edge features in the image are extracted and enhanced by using a smaller receptive field through convolution operations.

[0018] Furthermore, the low-bit images rotated by 0 degrees, 90 degrees, 180 degrees, and 270 degrees are subjected to convolution operations primarily using the first type of low-bit convolution kernel to generate corresponding outputs; the low-bit images rotated by 0 degrees, 90 degrees, 180 degrees, and 270 degrees are subjected to convolution operations primarily using the second type of low-bit convolution kernel to generate corresponding outputs.

[0019] The two output results are respectively rotated in the opposite direction by 0 degrees, 90 degrees, 180 degrees and 270 degrees. The average of the eight results after the reverse rotation is taken as the second average, which is the low bit edge enhancement image.

[0020] Furthermore, in step S3, the second sub-neural network fully mines the features of the edge region and flat region in the first feature image through convolution operation, extracts edge information from different angles in the first feature image, enhances the second sub-neural network's ability to perceive the boundary and contour of the first feature image, and obtains the second feature image through convolution operation.

[0021] Furthermore, in step S3, for the first feature image, the second sub-neural network uses three different shaped feature convolution kernels; the first feature convolution kernel takes four adjacent pixels in the same row; the second feature convolution kernel takes four adjacent pixels on the diagonal; the third feature convolution kernel takes four pixels in two adjacent rows and two columns; the receptive field of each feature convolution kernel is 1*4.

[0022] Furthermore, in step S3, the first feature image is rotated by 0 degrees, 90 degrees, 180 degrees and 270 degrees and then used as a new input to perform convolution operation through three feature convolution kernels of different shapes. The above operation results are then rotated in the opposite direction by 0 degrees, 90 degrees, 180 degrees and 270 degrees respectively, and the average value of the twelve results after the reverse rotation is used as the second feature image.

[0023] Furthermore, in step S4, the target image is initially a blank image of size H*W, containing only pixel position information and no pixel value information.

[0024] The location of any selected pixel in the target image is mapped to the corresponding location in the first feature image. Four pixels in the neighborhood of the corresponding point are taken as interpolation points. The output pixel value is obtained by weighted averaging of the four pixels and their respective weights. The output pixel value is then assigned to the selected pixel in the target image, thus providing the selected pixel with pixel value information. The weights are derived from the second feature image.

[0025] Using the same method, all pixels in the target image are traversed synchronously to obtain the pixel values ​​of all pixels in the target image. The target image, after being assigned pixel values, becomes the high-resolution image.

[0026] Furthermore, the low-resolution image is input into the trained neural network, which outputs the high-resolution image; the neural network is a convolutional neural network; the neural network includes a first sub-neural network and a second sub-neural network.

[0027] Furthermore, during the training process of the neural network, the training set consists of hundreds or thousands of data pairs, each data pair consisting of a low-resolution image of size h*w and a label image of size H*W; the input of the neural network is the low-resolution image, and the output of the network is the reconstructed high-resolution image. The mean squared error function is used as the loss function. During the iterative training of the network, the reconstructed high-resolution image gradually approaches the label image, and finally the model parameters with the smallest loss function are selected as the final neural network model.

[0028] Furthermore, when the first sub-neural network transforms the lookup table, since the low-resolution image is decomposed into the high-bit image and the low-bit image using high-low bit decomposition, the size of the lookup table is reduced without sampling.

[0029] When the second sub-neural network transforms the lookup table, it uses non-uniform sampling to adaptively select fewer sampling points that are more conducive to fitting the original distribution characteristics. Through mathematical analysis of the Gaussian model and data clustering, the 18 lookup tables are clustered into 7, reducing the size of the lookup tables.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] This invention provides a super-resolution method, comprising: S1, decomposing a low-resolution image of size h*w into a low-bit image of size h*w and a high-bit image of size h*w through a high-low bit decomposition operation; S2, obtaining a first feature image of size h*w with edge enhancement after feature extraction and enhancement of the high-bit image and the low-bit image through a first sub-neural network; S3, obtaining a second feature image of size h*w that fully reflects the high-frequency and low-frequency information of the image after passing the first feature image through a second sub-neural network; S4, reconstructing a high-resolution image of size H*W by interpolation of the first feature image, the second feature image, and the target image using a two-dimensional Gaussian distribution as the interpolation kernel; the target image is a blank image before the high-resolution image is assigned pixel values.

[0032] This invention provides a super-resolution method based on a combination of Gaussian models and deep learning. Compared to traditional methods, this invention achieves significantly better results with only a slight increase in power consumption, resulting in much clearer images. In terms of circuit structure, it requires no special computing units (NPU, TPU, etc.), achieving high performance at a low cost. The high-resolution images obtained using this invention achieve state-of-the-art SSIM (Structural Similarity Index) and PSNR (Peak Signal-to-Noise Ratio) metrics; furthermore, while achieving advanced image metrics and clear images, this invention significantly reduces the required memory. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the super-resolution method according to an embodiment of the present invention.

[0034] Figure 2 This is a schematic diagram of the super-resolution method according to an embodiment of the present invention.

[0035] Figure 3 This is a schematic diagram of three high-bit-level convolution kernels in the super-resolution method of this invention.

[0036] Figure 4 This is a schematic diagram of two low-bit convolution kernels in the super-resolution method of this invention.

[0037] Figure 5 A schematic diagram of three feature convolution kernels in the super-resolution method of this invention.

[0038] Figure 6 The label image in the super-resolution method of this invention.

[0039] Figure 7 This is a schematic diagram illustrating an advanced lookup table-based method for obtaining high-resolution images.

[0040] Figure 8 To obtain high-resolution images using conventional methods (such as Bicubic).

[0041] Figure 9 This is a schematic diagram illustrating the high-resolution image obtained by the method of the present invention.

[0042] Figure 10 A comparison chart of parameter indices for high-resolution images obtained by different methods.

[0043] The reference numerals in the attached figures are as follows:

[0044] 11-First high-bit sub-neural network; 12-First low-bit sub-neural network; 20-Second sub-neural network; 111-First high-bit convolution kernel; 112-Second high-bit convolution kernel; 113-Third high-bit convolution kernel; 121-First low-bit convolution kernel; 122-Second low-bit convolution kernel; 201-First feature convolution kernel; 202-Second feature convolution kernel; 203-Third feature convolution kernel. Detailed Implementation

[0045] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The advantages and features of the present invention will become clearer from the following description. It should be noted that the drawings are in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the illustration of the embodiments of the present invention.

[0046] For ease of description, some embodiments of this application may use spatially relative terms such as “above,” “below,” “top,” and “under” to describe the relationship between one element or component and another (or more) elements or components as shown in the accompanying drawings. It should be understood that, in addition to the orientations described in the drawings, spatially relative terms are also intended to include different orientations of the device during use or operation. For example, if the device in the drawings is flipped, it is described as an element or component “below” or “under” other elements or components, and will subsequently be positioned “above” or “on” other elements or components. The terms “first,” “second,” etc., used below are used to distinguish between similar elements and are not necessarily used to describe a particular order or temporal sequence. It should be understood that these terms, as used, may be replaced where appropriate.

[0047] This invention provides a super-resolution method, such as... Figure 1 and Figure 2 As shown, it includes:

[0048] S1. Decompose the low-resolution image LR of size h*w into a low-bit image LSB of size h*w and a high-bit image MSB of size h*w using a high-low bit decomposition operation;

[0049] S2, the high-bit image MSB and the low-bit image LSB are processed by the first sub-neural network for feature extraction and enhancement to obtain a first feature image with size h*w for edge enhancement;

[0050] S3. After the first feature image passes through the second sub-neural network 20, a second feature image of size h*w is obtained, which fully reflects the high-frequency and low-frequency information of the image.

[0051] S4. The first feature image, the second feature image, and the target image are interpolated using a two-dimensional Gaussian distribution as the interpolation kernel to reconstruct a high-resolution image HR of size H*W; the target image is the blank image before the high-resolution image is assigned pixel values.

[0052] The steps of the super-resolution method according to embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0053] The super-resolution method of this invention takes a low-resolution image (LR) as input to a trained neural network and outputs a high-resolution image (HR). The neural network is, for example, a convolutional neural network (CNN).

[0054] The neural network comprises a first sub-neural network and a second sub-neural network. During neural network training, the training set can consist of hundreds or thousands of data pairs, each pair consisting of a low-resolution image of size h*w and a label image of size H*W. The neural network input is the low-resolution image, and the network output is the reconstructed high-resolution image. The mean squared error function is used as the loss function. During iterative training, the reconstructed high-resolution image gradually approximates the label image, and finally, the model parameters with the minimum loss function are selected as the final neural network model.

[0055] Step S1: Decompose the low-resolution image LR of size h*w into a low-bit image LSB of size h*w and a high-bit image MSB of size h*w using a high-low bit decomposition operation. For example, to find the high n bits and low (8-n) bits of the 8-bit low-resolution image LR, this high-low bit decomposition operation can reduce the size of the transformation lookup table in the subsequent transformation lookup table stage without sampling. The low-resolution image LR can be 8 bits, 16 bits, or higher; there are no restrictions.

[0056] In step S2, the first sub-neural network includes a first high-bit-level sub-neural network 11 and a first low-bit-level sub-neural network 12. The first high-bit-level sub-neural network 11 extracts and enhances the high-frequency edge information contained in the MSB of the high-bit-level image to obtain a high-bit-level edge-enhanced image; the first low-bit-level sub-neural network 12 extracts and enhances the high-frequency edge information contained in the low-bit-level image to obtain a low-bit-level edge-enhanced image. The pixel values ​​at all the same pixel positions in the high-bit-level edge-enhanced image and the low-bit-level edge-enhanced image are added together to obtain the first feature image.

[0057] like Figure 2 and Figure 3 As shown, the high-bit-level image MSB can reflect the contour information of the original low-resolution image LR to a certain extent; the low-bit-level image LSB can reflect the detail information of the original low-resolution image LR to a certain extent. Based on their different characteristics, prior knowledge from image processing is incorporated into both high-bit-level and low-bit-level images, and convolutional kernels of different shapes are designed to better distinguish the features of edges at different angles in the image, thereby achieving efficient feature extraction and enhancement. For the high-bit-level image MSB, the first high-bit-level sub-neural network 11 can use three different shapes of high-bit-level convolutional kernels; the first type of high-bit-level convolutional kernel 111 takes three adjacent pixels in the same row; the second type of high-bit-level convolutional kernel 112 takes three adjacent pixels on the diagonal; and the third type of high-bit-level convolutional kernel 113 takes three adjacent pixels in different rows and columns; the receptive field of each high-bit-level convolutional kernel is 1*3.

[0058] The first high-bit sub-neural network 11 includes high-bit convolutional layers, several 1x1 convolutional layers, and activation layers. The high-bit convolutional layers include, for example, a first high-bit convolutional kernel 111, a second high-bit convolutional kernel 112, and a third high-bit convolutional kernel 113, all of which are 1x3 convolutions. The several 1x1 convolutional layers achieve information interaction and dimensionality scaling between channels with minimal parameter count. The activation layers can use the ReLU activation function for feature activation processing.

[0059] like Figure 2 and Figure 4As shown, for the low-bit image LSB, the first low-bit sub-neural network 12 employs two different shapes of low-bit convolutional kernels. The first type of low-bit convolutional kernel 121 takes two adjacent pixels located in the same row; the second type of low-bit convolutional kernel 122 takes two adjacent pixels located on the diagonal. The receptive field of each low-bit convolutional kernel is 1*2. The lower the receptive field of the low-bit convolutional kernel, the smaller the size of the generated lookup table during the transformation lookup table stage. High-frequency edge features in the image are extracted and enhanced using the smaller receptive field through convolution operations. The pixel values ​​at all the same pixel positions in the high-bit edge enhancement image and the low-bit edge enhancement image are added together to obtain the first feature image. The first feature image is an image rich in detail with edge enhancement. The first low-bit sub-neural network 12 includes low-bit convolutional layers, several 1x1 convolutional layers, and activation layers. Low-bit convolutional layers include, for example, a first low-bit convolutional kernel 121 and a second low-bit convolutional kernel 122; both types of high-bit convolutional kernels are 1x2 convolutions. Several 1x1 convolutional layers achieve information exchange between channels and dimensionality increase / decrease with a minimal number of parameters. Activation layers can use the ReLU activation function for feature activation processing.

[0060] When performing convolution operations on the high-bit image MSB and the low-bit image LSB, to increase the network's receptive field and improve its feature representation ability without increasing the lookup table size, the MSB and LSB are rotated by 0, 90, 180, and 270 degrees respectively and used as new inputs for convolution operations. The results are then rotated in the opposite direction by 0, 90, 180, and 270 degrees respectively, and then weighted and averaged to obtain the final output. The rotation can be clockwise or counterclockwise.

[0061] For example, the high-bit image MSB is rotated by 0 degrees, 90 degrees, 180 degrees, and 270 degrees, and then used as new input. This is then convolved with three sets of high-bit convolution kernels of different shapes. The results of these operations are then rotated in the opposite direction by 0 degrees, 90 degrees, 180 degrees, and 270 degrees, respectively. The average of the twelve results after the reverse rotation is taken as the first average, i.e., the high-bit edge enhancement image. Similarly, the low-bit image LSB is rotated by 0 degrees, 90 degrees, 180 degrees, and 270 degrees, and then used as new input. This is then convolved with two sets of low-bit convolution kernels of different shapes. The results of these operations are then rotated in the opposite direction by 0 degrees, 90 degrees, 180 degrees, and 270 degrees, respectively. The average of the eight results after the reverse rotation is taken as the second average, i.e., the low-bit edge enhancement image. The sum of the first and second averages is taken as the first feature image.

[0062] like Figure 2 and Figure 5As shown, in step S3, for the first feature image, the second sub-neural network 20 employs three different shaped feature convolution kernels. The first feature convolution kernel 201 takes four adjacent pixels in the same row; the second feature convolution kernel 202 takes four adjacent pixels on the diagonal; and the third feature convolution kernel 203 takes four pixels in two adjacent rows and two columns. The receptive field of each feature convolution kernel is 1*4. The second sub-neural network 20 includes feature convolution layers, several 1x1 convolution layers, and activation layers. The feature convolution layers include, for example, the first feature convolution kernel 201, the second feature convolution kernel 202, and the third feature convolution kernel 203; all three feature convolution kernels are 1x4 convolutions. Several 1x1 convolution layers achieve information interaction and dimensionality increase / decrease between channels with the fewest parameters. The activation layers can use the ReLU activation function for feature activation processing.

[0063] When performing convolution operations on the first feature image, to increase the network's receptive field and improve its feature representation ability without increasing the lookup table size, the first feature image is rotated by 0 degrees, 90 degrees, 180 degrees, and 270 degrees and then used as new input for convolution operations. The results of these operations are then rotated in the opposite direction by 0 degrees, 90 degrees, 180 degrees, and 270 degrees respectively, and then weighted and averaged to obtain the second feature image. The rotation can be clockwise or counterclockwise. For example, the first feature image can be rotated by 0 degrees, 90 degrees, 180 degrees, and 270 degrees and then used as new input for convolution operations with three sets of feature convolution kernels of different shapes. The results of these operations are then rotated in the opposite direction by 0 degrees, 90 degrees, 180 degrees, and 270 degrees respectively, and the average of the twelve results after the reverse rotation is taken as the second feature image.

[0064] In step S3, the second sub-neural network 20 fully mines the features of the edge region and flat region in the first feature image through convolution operation, extracts edge information from different angles in the first feature image, enhances the second sub-neural network's ability to perceive the boundary and contour of the first feature image, and obtains a second feature image that can reflect the high-frequency and low-frequency information features of the image through convolution operation.

[0065] like Figure 2 As shown, in step S4, the first feature image, the second feature image, and the target image are interpolated using a two-dimensional Gaussian distribution as the interpolation kernel to reconstruct a high-resolution image HR of size H*W; the target image is the blank image before the pixel values ​​are assigned to the high-resolution image. The reconstructed high-resolution image HR of size H*W is generated using a weighted interpolation function. This weighted interpolation function takes the first feature image as input, calculates the weights using the second feature image based on a Gaussian distribution, and outputs the reconstructed high-resolution image HR of size H*W.

[0066] The initial target image is a blank image of size H*W, containing only pixel location information but no pixel value information. The location of any selected pixel in the target image is mapped to its corresponding location in the first feature image. Four neighboring pixels of this location are taken as interpolation points. A weighted average of these four pixels with their respective weights is used to obtain the output pixel value. This output pixel value is then assigned to the selected pixel in the target image, thus providing pixel value information. The weights are derived from the second feature image. Using the same method, all pixels in the target image are traversed synchronously to obtain the pixel values ​​for all pixels. The target image, after being assigned pixel values, becomes a high-resolution image.

[0067] Specifically, the target image is the blank image before the high-resolution image is assigned pixel values. For any pixel in the high-resolution image... That is, any pixel selected in the target image. pixel The corresponding point mapped to the first feature image is .

[0068] Corresponding points The method for calculating location information is as follows , ;

[0069] The four pixels in the neighborhood of the corresponding point , , , The selection method is as follows: This is for rounding up.

[0070]

[0071]

[0072]

[0073]

[0074] The weighted weights for each of the four pixels are as follows: , , , Calculated using a two-dimensional Gaussian distribution model:

[0075]

[0076] by , , represent Variance of direction represent Variance of direction represent direction and The correlation coefficient between directions, in this method, will be the corresponding value in the second feature image that reflects the high-frequency and low-frequency information of the image. Approximately , , , To perform the absolute value operation, substitute the values ​​into the above formula to calculate the result. . , , The calculation method is the same.

[0077] The output pixel value is calculated using a weighted average method. :

[0078]

[0079] Output pixel values By assigning pixel values ​​to selected pixels in the target image, the selected pixels acquire pixel value information; the weights are derived based on the second feature image; using the same method, all pixels in the target image are traversed synchronously to obtain the pixel values ​​of all pixels in the target image, and the target image, after being assigned pixel values, becomes a high-resolution image.

[0080] Lookup tables are a commonly used data structure in computer science and digital systems that optimizes computational performance by pre-compiling and storing results. The core idea of ​​lookup tables is to pre-compute and store all possible results; when a specific result is needed, it is looked up directly through an index instead of being calculated in real time. Lookup tables are an important technique for performance optimization, particularly suitable for computationally intensive scenarios with limited input ranges. They significantly improve computation speed (trading space for time) and reduce the processor burden on complex calculations.

[0081] The method for obtaining (converting) the lookup table of the present invention includes: traversing all combinations of inputs and combining them with the corresponding network outputs to obtain a lookup table. During inference, the lookup table can replace the convolution operations related to the neural network.

[0082] For example, an 8-bit low-resolution image LR of size h*w is decomposed into a low 4-bit image LSB of size h*w and a high 4-bit image MSB of size h*w through a high-low bit decomposition operation. The low-resolution image LR can be 8 bits, 16 bits, or higher bits, without limitation.

[0083] When the first sub-neural network transforms the lookup table, the low-resolution image is decomposed into a high-bit image and a low-bit image by using high-low bit decomposition, which reduces the size of the lookup table without sampling. For example, by using high-low bit decomposition, the input of the 8-bit low-resolution image LR is split into two 4-bit inputs (low 4-bit image LSB and high 4-bit image MSB), which reduces the size of the lookup table without sampling.

[0084] When the second sub-neural network 20 transforms the lookup table, it uses non-uniform sampling and the model adaptively selects fewer sampling points that are more conducive to fitting the original distribution characteristics. Through mathematical analysis of the Gaussian model and data clustering, the 18 lookup tables are clustered into 7, reducing the size of the lookup tables.

[0085] When the first sub-neural network transforms the lookup table, such as Figure 3 As shown, the process of generating corresponding outputs by performing convolution operations primarily using a first type of high-bit-level convolution kernel 111 on high-bit-level images (MSB) rotated by 0 degrees, 90 degrees, 180 degrees, and 270 degrees, is defined as the first model. The process of generating corresponding outputs by performing convolution operations primarily using a second type of high-bit-level convolution kernel 112 on high-bit-level images (MSB) rotated by 0 degrees, 90 degrees, 180 degrees, and 270 degrees, is defined as the second model. The process of generating corresponding outputs by performing convolution operations primarily using a third type of high-bit-level convolution kernel 113 on high-bit-level images (MSB) rotated by 0 degrees, 90 degrees, 180 degrees, and 270 degrees, is defined as the third model. In the first sub-neural network transformation lookup table process, for the processing of high-bit image MSB, each model's input contains three 4-bit data, and each 4-bit data has 16*16*16 different combinations. The network output corresponding to each combination requires one byte of memory for storage. Therefore, the table storage for the three models requires 3*16*16*16 = 12288 bytes = 12KB.

[0086] Similarly, when the first sub-neural network transforms the lookup table, the processing of low-bit image LSBs, such as... Figure 4As shown, the process of generating corresponding outputs by performing convolution operations primarily using the first type of low-bit convolution kernel 121 on low-bit image LSBs rotated by 0 degrees, 90 degrees, 180 degrees, and 270 degrees, is defined as one model. The process of generating corresponding outputs by performing convolution operations primarily using the second type of low-bit convolution kernel 122 on low-bit image LSBs rotated by 0 degrees, 90 degrees, 180 degrees, and 270 degrees, is defined as another model. There are two models for processing low-bit image LSBs. For low-bit image LSB processing, each model's input contains two 4-bit data points. Each 4-bit data point has 16*16 different combinations. The network output corresponding to each combination requires one byte of memory for storage. Therefore, the table storage for the two models requires 2*16*16 = 512 bytes = 0.5KB.

[0087] When the second sub-neural network 20 transforms the lookup table, such as Figure 5 As shown, the first feature images rotated by 0 degrees and 180 degrees are processed by convolution operations using the first feature convolution kernel 201 to produce the corresponding output. ( represent The process of (variance of direction) is defined as Model 1. The first feature images rotated by 0 degrees and 180 degrees are subjected to convolution operations primarily using the first feature convolution kernel 201 to produce the corresponding output. ( represent The process of (variance of direction) is defined as Model 2. The first feature images rotated by 0 degrees and 180 degrees are subjected to convolution operations primarily using the first feature convolution kernel 201 to produce the corresponding output. ( represent direction and The process of determining the correlation coefficient between directions is defined as Model 3. The first feature images rotated by 90 degrees and 270 degrees are then subjected to convolution operations primarily using the first feature convolution kernel 201 to produce the corresponding outputs. The process is defined as Model 4. The first feature images, rotated by 90 degrees and 270 degrees, are subjected to convolution operations primarily using the first feature convolution kernel 201 to produce the corresponding output. The process is defined as Model 5. The first feature images, rotated by 90 degrees and 270 degrees, are subjected to convolution operations primarily using the first feature convolution kernel 201 to produce the corresponding output. The process is defined as Model Six. Similarly, for the second feature convolution kernel 202 and the third feature convolution kernel 203, each feature convolution kernel can generate 6 models, and the three feature convolution kernels can generate 18 models.

[0088] Through analysis of the two-dimensional Gaussian distribution model, different models... , , They are somewhat related, specifically.

[0089] ,

[0090] ,

[0091] ,

[0092] ,

[0093] ;

[0094] For the first type of feature, 201 pairs of rotating convolution kernels are used. ; For the second type of feature, 202 pairs of convolution kernels are rotated. ; The third type of feature convolution kernel has 203 rotations. . Three types of feature convolution kernels are rotated respectively. . Three types of feature convolution kernels are rotated respectively. Therefore, clustering several identical models among the 18 models into one model reduces the number of models from 18 to 7, which means clustering 18 lookup tables into 7.

[0095] In the second sub-neural network 20, during the transformation of the lookup table, for the processing of the first feature image, each model's input contains four 8-bit data points. During the lookup table transformation, a non-uniform sampling method is used. The interval distance and interval points are adaptively learned by the network, sampling the 256 points represented by the 8-bit data into 7 points. Compared with the conventional method of equal-distance sampling, this method can better fit the original data distribution with fewer data points. The values ​​of unsampled points are calculated by interpolation of the sampled points in the neighborhood. Each model's input contains four sampled data points, and each data point has 7*7*7*7 = 2401 different combinations. Each combination's corresponding network output requires one byte of memory for storage. Therefore, the seven model inputs require 7*7*7*7*7 = 16804 bytes = 16.4KB. Thus, the entire neural network transformation into a lookup table requires a total of 12KB + 0.5KB + 16.4KB = 28.9KB of memory.

[0096] Figure 6 For label images. Figure 7 A schematic diagram illustrating the high-resolution image obtained using the previous method. Figure 8To obtain high-resolution images using conventional methods (such as Bicubic). Figure 9 This is a schematic diagram illustrating the high-resolution image obtained by the method of the present invention. Figure 10 A comparison chart showing the parameter indices of high-resolution images obtained using different methods. For example... Figures 6 to 10 As shown, the SSIM (Structural Similarity Index) and PSNR (Peak Signal-to-Noise Ratio) indices of the high-resolution images obtained using the super-resolution method of this invention have reached advanced levels. Furthermore, based on the advanced level of image indices and the clarity of the images, the method of this invention significantly reduces memory usage. For example, the memory usage of the method of this invention is 28.9K Bytes, while the memory usage of the method before the improvement is 1.67M. The memory required by the method before the improvement is 58 times that of this invention.

[0097] SSIM is a metric used to measure the similarity between two images, and it is widely used in image quality assessment, compression effect analysis, image reconstruction, and other fields. Based on the characteristics of the human visual system, SSIM comprehensively considers the brightness, contrast, and structural information of an image, making it more consistent with human subjective perception. A higher SSIM value indicates a better reconstruction result.

[0098] PSNR is calculated based on mean squared error and represents the ratio of the maximum possible power (peak power) of the signal to the noise power. It is primarily used to assess the degree of image quality degradation before and after image processing. It compares the difference between the original image and the reconstructed image; a higher PSNR value indicates a better reconstruction result.

[0099] In summary, a super-resolution method includes: S1, decomposing a low-resolution image of size h*w into a low-bit image of size h*w and a high-bit image of size h*w through a high-low bit decomposition operation; S2, after feature extraction and enhancement of the high-bit image and the low-bit image through a first sub-neural network, a first feature image of size h*w with edge enhancement is obtained; S3, after the first feature image is processed by a second sub-neural network, a second feature image of size h*w that fully reflects the high-frequency and low-frequency information of the image is obtained; S4, the first feature image, the second feature image, and the target image are interpolated using a two-dimensional Gaussian distribution as the interpolation kernel to reconstruct a high-resolution image of size H*W; the target image is the blank image before the high-resolution image is assigned pixel values. This invention is a super-resolution method based on a combination of Gaussian model and deep learning. Compared with traditional methods, this invention achieves a much better result with a slight increase in power consumption, resulting in a clearer image. In terms of circuit structure, no special computing unit (NPU, TPU, etc.) is required, achieving high performance at a low cost. The high-resolution images obtained using the super-resolution method of this invention achieve state-of-the-art SSIM (Structural Similarity Index) and PSNR (Peak Signal-to-Noise Ratio) metrics. Furthermore, while achieving advanced image metrics and clear images, this method significantly reduces the required memory. Super-resolution technology is implemented at low cost and can be implemented in ordinary ISPs and DDIC chips. Iterative updates of the scheme are achieved by updating a lookup table, requiring no changes to the hardware architecture and offering high flexibility.

[0100] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. The methods disclosed in the embodiments are described simply because they correspond to the devices disclosed in the embodiments; relevant details can be found in the method section.

[0101] The above description is merely a description of preferred embodiments of the present invention and is not intended to limit the scope of the present invention. Any person skilled in the art can make possible changes and modifications to the technical solutions of the present invention by utilizing the methods and techniques disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the protection scope of the technical solutions of the present invention.

Claims

1. A super-resolution method, characterized in that, include: S1. Decompose the low-resolution image of size h*w into a low-bit image of size h*w and a high-bit image of size h*w using a high-low bit decomposition operation; S2. The high-bit image and the low-bit image are processed by the first sub-neural network to extract and enhance features, resulting in a first feature image with edge enhancement of size h*w. S3. After the first feature image is processed by the second sub-neural network, a second feature image of size h*w is obtained, which fully reflects the high-frequency and low-frequency information of the image. S4. The first feature image, the second feature image, and the target image are interpolated using a two-dimensional Gaussian distribution as the interpolation kernel to reconstruct a high-resolution image of size H*W; the target image is the blank image before the high-resolution image is assigned pixel values.

2. The super-resolution method as described in claim 1, characterized in that, In step S2, the first sub-neural network includes a first high-bit sub-neural network and a first low-bit sub-neural network; The first high-bit sub-neural network extracts and enhances the high-frequency edge information contained in the high-bit image to obtain a high-bit edge enhanced image; the first low-bit sub-neural network extracts and enhances the high-frequency edge information contained in the low-bit image to obtain a low-bit edge enhanced image. The first feature image is obtained by adding the pixel values ​​at all the same pixel positions in the high-bit edge enhancement image and the low-bit edge enhancement image.

3. The super-resolution method as described in claim 2, characterized in that, For the high-bit image and the low-bit image, prior knowledge from known image processing is added to the first sub-neural network. Through convolution operations, the first sub-neural network model can better distinguish the features of edges at different angles in the image, thus completing the extraction and enhancement of high-frequency edge features.

4. The super-resolution method as described in claim 2, characterized in that, For the high-bit image, the first high-bit sub-neural network uses three different shapes of high-bit convolution kernels; the first type of high-bit convolution kernel takes three adjacent pixels in the same row; the second type of high-bit convolution kernel takes three adjacent pixels on the diagonal; and the third type of high-bit convolution kernel takes three adjacent pixels in different rows and columns; the receptive field of each type of high-bit convolution kernel is 1*3.

5. The super-resolution method as described in claim 4, characterized in that, The high-bit images rotated by 0 degrees, 90 degrees, 180 degrees, and 270 degrees are processed through convolution operations primarily using the first type of high-bit convolution kernel to produce corresponding outputs; the high-bit images rotated by 0 degrees, 90 degrees, 180 degrees, and 270 degrees are processed through convolution operations primarily using the second type of high-bit convolution kernel to produce corresponding outputs; the high-bit images rotated by 0 degrees, 90 degrees, 180 degrees, and 270 degrees are processed through convolution operations primarily using the third type of high-bit convolution kernel to produce corresponding outputs; The three outputs described above are rotated in the opposite direction by 0 degrees, 90 degrees, 180 degrees and 270 degrees respectively. The average of the twelve results after the reverse rotation is taken as the first average, which is the high bit edge enhancement image.

6. The super-resolution method as described in claim 2, characterized in that, For the low-bit image, the first low-bit sub-neural network uses two different shapes of low-bit convolution kernels; the first low-bit convolution kernel takes two adjacent pixels in the same row; the second low-bit convolution kernel takes two adjacent pixels on the diagonal; the receptive field of each low-bit convolution kernel is 1*2; the high-frequency edge features in the image are extracted and enhanced by using a smaller receptive field through convolution operation.

7. The super-resolution method as described in claim 6, characterized in that, The low-bit images rotated by 0 degrees, 90 degrees, 180 degrees, and 270 degrees are processed through convolution operations primarily using the first type of low-bit convolution kernel to produce corresponding outputs; the low-bit images rotated by 0 degrees, 90 degrees, 180 degrees, and 270 degrees are processed through convolution operations primarily using the second type of low-bit convolution kernel to produce corresponding outputs. The two outputs described above are rotated in the opposite direction by 0 degrees, 90 degrees, 180 degrees and 270 degrees respectively. The average of the eight results after the reverse rotation is taken as the second average, which is the low-bit edge enhancement image.

8. The super-resolution method as described in claim 1, characterized in that, In step S3, the second sub-neural network fully mines the features of the edge region and flat region in the first feature image through convolution operation, extracts edge information from different angles in the first feature image, enhances the second sub-neural network's ability to perceive the boundary and contour of the first feature image, and obtains the second feature image through convolution operation.

9. The super-resolution method as described in claim 1, characterized in that, In step S3, for the first feature image, the second sub-neural network uses three different shaped feature convolution kernels; the first feature convolution kernel takes four adjacent pixels in the same row; the second feature convolution kernel takes four adjacent pixels on the diagonal; the third feature convolution kernel takes four pixels in two adjacent rows and two columns; the receptive field of each feature convolution kernel is 1*4.

10. The super-resolution method as described in claim 9, characterized in that, In step S3, the first feature image is rotated by 0 degrees, 90 degrees, 180 degrees and 270 degrees and then used as a new input to perform convolution operation through the feature convolution kernels of three different shapes. The above operation results are rotated in the opposite direction by 0 degrees, 90 degrees, 180 degrees and 270 degrees respectively. The average value of the twelve results after the reverse rotation is used as the second feature image.

11. The super-resolution method as described in claim 1, characterized in that, In step S4, the target image is initially a blank image of size H*W, containing only pixel position information and no pixel value information; The location of any selected pixel in the target image is mapped to the corresponding location in the first feature image. Four pixels in the neighborhood of the corresponding point are taken as interpolation points. The output pixel value is obtained by weighted averaging of the four pixels and their respective weights. The output pixel value is then assigned to the selected pixel in the target image, thus providing the selected pixel with pixel value information. The weights are derived from the second feature image. Using the same method, all pixels in the target image are traversed synchronously to obtain the pixel values ​​of all pixels in the target image. The target image, after being assigned pixel values, becomes the high-resolution image.

12. The super-resolution method as described in claim 1, characterized in that, The low-resolution image is input into the trained neural network, which outputs the high-resolution image; the neural network is a convolutional neural network; the neural network includes a first sub-neural network and a second sub-neural network.

13. The super-resolution method as described in claim 12, characterized in that, During the training of the neural network, the training set consists of hundreds or thousands of data pairs. Each data pair consists of a low-resolution image of size h*w and a label image of size H*W. The input of the neural network is the low-resolution image, and the output of the network is the reconstructed high-resolution image. The mean squared error function is used as the loss function. During the iterative training of the network, the reconstructed high-resolution image gradually approaches the label image. Finally, the model parameters with the smallest loss function are selected as the final neural network model.

14. The super-resolution method as described in claim 12, characterized in that, When the first sub-neural network transforms the lookup table, the low-resolution image is decomposed into the high-bit image and the low-bit image using high-low bit decomposition, thereby reducing the size of the lookup table without sampling. When the second sub-neural network transforms the lookup table, it uses non-uniform sampling to adaptively select fewer sampling points that are more conducive to fitting the original distribution characteristics. Through mathematical analysis of the Gaussian model and data clustering, the 18 lookup tables are clustered into 7, reducing the size of the lookup tables.

Images