Super-resolution method

Abstract

According to the super-resolution method provided by the invention, multi-scale super-resolution can be realized, and the method is not limited by decimal multiplying power; the processed image is converted from an RGB domain to a YUV domain, and only a Y channel is processed; a lookup table for image smoothing and a lookup table for image enhancement are obtained through AI model training, and the lookup tables occupy very small memory and only have dozens of K bytes. Respectively generating an image smoothing lookup table and an image enhancing lookup table in stages from a rough lookup table to an accurate lookup table; processing the data by stages to realize smoothing and enhancement of the image; and by utilizing singular value suppression, false colors are eliminated at extremely low cost. And a common chip can complete a super-resolution function without depending on a special computing unit such as an NPU (Neural Network Processor). According to the super-resolution method, a high-resolution clear image is finally obtained without depending on a special calculation unit, super-resolution is realized through low power consumption of a lookup table occupying a very small memory, and the cost is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a super-resolution method. Background Art

[0002] Super-Resolution (SR) is to improve the resolution of the original image through hardware or software methods. The process of obtaining a high-resolution image from a series of low-resolution images is super-resolution reconstruction; super-resolution converts low-resolution images or videos into high-resolution images or videos.

[0003] There are mainly two ways to achieve super-resolution: traditional and complex neural network methods. The traditional method has advantages such as simple logic, easy implementation, and low power consumption, but the effect is generally that the restored image is relatively blurred. When the product realizes the magnification function, the traditional method is basically adopted. The image restored by the complex neural network method is clearer, sharper, delicate and natural, but the logic is complex, the power consumption is high, and it requires special computing units such as NPU (Neural Network Processor) to support. Summary of the Invention

[0004] The purpose of the present invention is to provide a super-resolution method, which can finally obtain a high-resolution clear image, without relying on special computing units, and realize super-resolution with low power consumption through a lookup table that occupies very little memory, reducing costs.

[0005] The present invention provides a super-resolution method, including:

[0006] Step S1: Provide a low-resolution image, perform preprocessing of size adjustment and tone mapping on the low-resolution image to obtain a first RGB-domain image corresponding to the width and height of the target high-resolution image, and convert the first RGB-domain image into a first YUV-domain image, where the first YUV-domain image includes three-channel images of a Y channel, a U channel, and a V channel;

[0007] Step S2: Use a lookup table for image smoothing to perform image smoothing processing only on the Y-channel image;

[0008] Step S3: Perform image enhancement processing on the Y-channel image after image smoothing using a lookup table for image enhancement;

[0009] Step S4: Perform singular value suppression processing on the Y-channel image after image enhancement processing;

[0010] Step S5: Convert the second YUV-domain image composed of the U channel, the V channel, and the Y-channel image after singular value suppression processing into a high-resolution RGB-domain image;

[0011] Among them, both the look-up table for image smoothing and the look-up table for image enhancement are obtained through training with an AI model.

[0012] Further, in step S2, the method for obtaining the look-up table for image smoothing includes:

[0013] S21. Train a first network model for image smoothing. The first input image of the first network model is an edge non-smooth image with a size of W*H, and the first label image of the first network model is an edge smooth image with a size of W*H. After the training of the first network model converges, convert the mapping relationship between the first input image and the first label image into a rough look-up table for image smoothing;

[0014] S22. Perform parameter training on the rough look-up table for image smoothing. The first input image for training is an edge non-smooth image with a size of W*H, and the first label image for training is an edge smooth image with a size of W*H. After the parameter training converges, the obtained look-up table is an accurate look-up table for image smoothing; Use the accurate look-up table for image smoothing as the look-up table for image smoothing in step S2.

[0015] Further, in step S3, the method for obtaining the look-up table for image enhancement includes:

[0016] S31. Train a second network model for image enhancement. The second input image of the second network model is an edge blurred image with a size of W*H, and the second label image of the second network model is an edge sharp image with a size of W*H. After the training of the second network model converges, convert the mapping relationship between the second input image and the second label image into a rough look-up table for image enhancement;

[0017] S32. Perform parameter training on the rough look-up table for image enhancement. The second input image for training is an edge blurred image with a size of W*H, and the second label image for training is an edge sharp image with a size of W*H. After the training converges, the obtained look-up table is the accurate look-up table for image enhancement; Use the accurate look-up table for image enhancement as the look-up table for image enhancement in step S3.

[0018] Further, step S21 further includes: The first input image is obtained by using a non-linear operation to generate certain serrations and / or burrs on the image edge;

[0019] Step S31 further includes: The second input image is obtained by using a weighted calculation kernel with a blur effect to destroy the high-frequency details of the image.

[0020] Further, in step S21, the first label image is obtained by using a complex neural network to enhance the image edge details;

[0021] In step S31, the second tag image can be obtained by using a weighted calculation kernel with an enhancement effect and enhancing the high-frequency details of the image through the complex neural network.

[0022] Furthermore, both the first network model and the second network model include the dense structure in the AI model; the structures of both the first network model and the second network model include a number of convolutional layers and the activation layers corresponding to the convolutional layers respectively.

[0023] Furthermore, step S2 specifically includes: selecting any initial pixel in the Y-channel image, using the initial pixel as the center, and forming a first pixel window of m rows * n columns composed of the initial pixel and a plurality of neighboring pixels around the initial pixel; selecting at least two first sub-pixel windows including the initial pixel and several neighboring pixels with different orientations in the first pixel window; arranging all pixel values in each first sub-pixel window in a certain order as a first pixel index, querying the output corresponding to the lookup table for image smoothing, and taking the weighted average of the outputs corresponding to all first sub-pixel windows as the pixel value of the initial pixel after smoothing; traversing each initial pixel in the Y-channel image and using the same smoothing method to obtain the Y-channel image after image smoothing processing.

[0024] Furthermore, when storing the lookup table for image smoothing, sampling and storing the first pixel index at a specified interval; when the first sub-pixel window queries the lookup table for image smoothing, directly querying the smoothed pixel value for the first pixel index already stored in the lookup table for image smoothing; for the first pixel index not stored in the lookup table for image smoothing, interpolating and calculating by using the pixel indexes P1, P2... Pn closest to the first pixel index and the corresponding smoothed pixel values V1, V2... Vn already stored.

[0025] Further, step S3 specifically includes: selecting any to-be-processed pixel in the Y-channel image after the image smoothing process, using the to-be-processed pixel as the center, and forming a second pixel window of a rows * b columns composed of the to-be-processed pixel and a plurality of neighboring pixels around the to-be-processed pixel; selecting at least two second sub-pixel windows in the second pixel window, which include the processed pixel and several neighboring pixels with different orientations; arranging all the pixel values in each second sub-pixel window in a certain order as a second pixel index, querying the corresponding output of the lookup table for image enhancement, and taking the weighted average of the outputs corresponding to all the second sub-pixel windows as the pixel value of the to-be-processed pixel after enhancement; traversing each to-be-processed pixel in the Y-channel image after the image smoothing process, and obtaining the Y-channel image after the image enhancement process by using the same enhancement method.

[0026] When storing the lookup table for image enhancement, sample and store the second pixel index at a specified interval; when the second sub-pixel window queries the lookup table for image enhancement, directly query the enhanced pixel value for the second pixel index already stored in the lookup table for image enhancement; for the second pixel index not stored, calculate it by interpolation using the pixel indices L1, L2... Ln closest to the second pixel index and the corresponding enhanced pixel values Z1, Z2... Zn already stored in the lookup table for image enhancement.

[0027] Further, in step S4, the singular value suppression process for the Y-channel image after the image enhancement process specifically includes: selecting any to-be-processed pixel in the Y-channel image after the image enhancement process, using the to-be-processed pixel as the center, and forming a third pixel window of c rows * d columns composed of the to-be-processed pixel and a plurality of neighboring pixels around the to-be-processed pixel; among the c * d pixel values in the third pixel window, if a certain pixel value is significantly abnormal compared to other pixel values, then the pixel value is defined as a singular value, and the singular value is corrected to any one of the average value, median value, or weighted average value of the other several pixel values in the third pixel window; if there is no singular value, keep the original pixel values of the pixels in the third pixel window.

[0028] Traverse each to-be-processed pixel in the Y-channel image after the image enhancement process, and obtain the Y-channel image after the image singular value suppression process by using the same singular value suppression process method.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] The present invention provides a super-resolution method, including: Step S1, providing a low-resolution image, performing preprocessing of size adjustment and tone mapping on the low-resolution image to obtain a first RGB-domain image corresponding to the width and height of the target high-resolution image, and converting the first RGB-domain image into a first YUV-domain image, where the first YUV-domain image includes three-channel images of a Y channel, a U channel, and a V channel; Step S2, performing image smoothing processing only on the Y-channel image by using a lookup table for image smoothing; Step S3, performing image enhancement processing on the Y-channel image after image smoothing processing by using a lookup table for image enhancement; Step S4, performing singular value suppression processing on the Y-channel image after image enhancement processing; Step S5, converting a second YUV-domain image composed of the U channel, the V channel, and the Y-channel image after singular value suppression processing into a high-resolution RGB-domain image; wherein, both the lookup table for image smoothing and the lookup table for image enhancement are obtained through training of an AI model.

[0031] The present invention provides a super-resolution method, which can achieve multi-scale super-resolution and is not limited by fractional magnification factors; converts the processed image from the RGB domain to the YUV domain and only processes the Y channel; obtains the lookup table for image smoothing and the lookup table for image enhancement through training of an AI model, and the lookup tables occupy very little memory, only about a dozen kilobytes. Generates the lookup table for image smoothing and the lookup table for image enhancement in stages, from a rough lookup table to an accurate lookup table; processes data in stages to achieve image smoothing and enhancement; uses singular value suppression to eliminate false colors at extremely low cost. Does not rely on special computing units such as NPUs (neural network processors), and ordinary chips can complete the super-resolution function. The super-resolution method of the present invention finally obtains a high-resolution clear image, does not rely on special computing units, and realizes super-resolution with low power consumption through lookup tables that occupy very little memory, reducing costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a schematic flowchart of the super-resolution method according to an embodiment of the present invention.

[0033] Figure 2 is a flowchart of the super-resolution method according to an embodiment of the present invention.

[0034] Figure 3 is a schematic diagram comparing the RGB-domain image and the YUV-domain image in the super-resolution method according to an embodiment of the present invention.

[0035] Figure 4 is a schematic diagram of lookup table query in the super-resolution method according to an embodiment of the present invention.

[0036] Figure 5 is a schematic diagram of an example image before image smoothing in the super-resolution method according to an embodiment of the present invention.

[0037] Figure 6 is Figure 5 a schematic diagram of the smoothed example image.

[0038] Figure 7 is Figure 5 a comparison schematic diagram of the locally enlarged view within the red frame before smoothing intercepted from Figure 6 and the locally enlarged view within the red frame after smoothing intercepted from

[0039] Figure 8 is Figure 5 a comparison schematic diagram of the locally enlarged view within the blue frame before smoothing intercepted from Figure 6 and the locally enlarged view within the blue frame after smoothing intercepted from

[0040] Figure 9 is the first example diagram for comparing before and after singular value suppression of the Y-channel image after image enhancement processing.

[0041] Figure 10 is the second example diagram for comparing before and after singular value suppression of the Y-channel image after image enhancement processing.

[0042] Figure 11 is a comparison diagram of the processing effects of three kinds of super-resolution.

[0043] Figure 12 is a comparison diagram of the processing effects of multiple kinds of super-resolution. Detailed implementation manners

[0044] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. According to the following description, the advantages and features of the present invention will be clearer. It should be noted that the accompanying drawings are all in very simplified forms and use non-precise scales, and are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention.

[0045] In the accompanying drawings, for clarity, the dimensions of layers, regions, elements and their relative dimensions may be exaggerated. The same reference numerals represent the same elements throughout.

[0046] It should be understood that when an element or layer is referred to as being "on", "adjacent to", "connected to" or "coupled to" another element or layer, it can be directly on, adjacent to, connected or coupled to the other element or layer, or there may be intervening elements or layers. In contrast, when an element is referred to as being "directly on", "directly adjacent to", "directly connected to" or "directly coupled to" another element or layer, there are no intervening elements or layers. It should be understood that although the terms first, second, third, etc. may be used to describe various elements, components, regions, layers and / or parts, these elements, components, regions, layers and / or parts should not be limited by these terms. These terms are only used to distinguish one element, component, region, layer or part from another element, component, region, layer or part. Thus, without departing from the teachings of the present application, the first element, component, region, layer or part discussed below may be referred to as the second element, component, region, layer or part. And when discussing the second element, component, region, layer or part, it does not imply that there must be a first element, component, region, layer or part in the present application.

[0047] Spatial relationship terms such as "under", "below", "lower", "beneath", "above", "upper", etc. are used herein for convenience in describing the relationship of one element or feature shown in the figures to other elements or features. It should be understood that, in addition to the orientation shown in the figures, spatial relationship terms are intended to include different orientations of the device in use and operation. For example, if the device in the figures is flipped, then an element or feature described as "under" or "beneath" or "below" other elements or features will be oriented "above" the other elements or features. Thus, the exemplary terms "under" and "below" can include both an upper and a lower orientation. The device may be otherwise oriented (rotated 90 degrees or other orientations) and the spatial descriptors used herein are to be interpreted accordingly.

[0048] The purpose of the terms used herein is only to describe specific embodiments and is not a limitation of the present application. As used herein, the singular forms "a", "an" and "the" are also intended to include the plural forms unless the context clearly dictates otherwise. It should also be understood that the terms "comprising" and / or "including", when used in this specification, specify the presence of the stated features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups. As used herein, the term "and / or" includes any and all combinations of the related listed items.

[0049] Embodiments of the present invention provide a super-resolution method, as Figure 1 shown, including:

[0050] Step S1: Provide a low-resolution image, perform preprocessing of size adjustment and tone mapping on the low-resolution image to obtain a first RGB-domain image corresponding to the width and height of the target high-resolution image, and convert the first RGB-domain image into a first YUV-domain image. The first YUV-domain image includes three-channel images of a Y channel, a U channel, and a V channel;

[0051] Step S2: Use a lookup table for image smoothing to perform image smoothing processing only on the Y-channel image;

[0052] Step S3: Use a lookup table for image enhancement to perform image enhancement processing on the Y-channel image after image smoothing;

[0053] Step S4: Perform singular value suppression processing on the Y-channel image after image enhancement;

[0054] Step S5: Convert the second YUV-domain image composed of the U channel, the V channel, and the Y-channel image after singular value suppression into a high-resolution RGB-domain image;

[0055] Among them, both the lookup table for image smoothing and the lookup table for image enhancement are obtained through AI model training. The high-resolution RGB-domain image is the target high-resolution image.

[0056] The following details each step of the super-resolution method according to the embodiments of the present invention with reference to the accompanying drawings.

[0057] Step S1: Provide a low-resolution image, perform preprocessing of size adjustment and tone mapping on the low-resolution image through an application processor to obtain a first RGB-domain image with a preset width W and height H, and convert the first RGB-domain image into a first YUV-domain image. The first YUV-domain image includes three-channel images of a Y channel, a U channel, and a V channel.

[0058] The application processor is the core computing unit of an intelligent device, executing application programs and performing graphics processing. Tone mapping is mainly used to convert a high dynamic range (HDR) image into a low dynamic range (LDR) image. Tone mapping solves the following problems: The brightness range contained in the HDR image far exceeds the capabilities of a standard display device; it is necessary to retain visually important details and contrast; avoid the image looking too dark or overexposed. Tone mapping maintains a natural appearance, avoids halo artifacts, retains details while compressing the dynamic range, and meets the requirements of real-time processing. Tone mapping algorithms usually need to find a balance between retaining details, maintaining contrast, and generating a natural appearance.

[0059] The memory size of the lookup table corresponding to the YUV-domain image is about 1 / 3 of the memory size of the lookup table corresponding to the RGB-domain image. Converting the first RGB-domain image into the first YUV-domain image can save the memory of the subsequent corresponding lookup table for image smoothing and the lookup table for image enhancement; moreover, asFigure 3 As shown in the figure, color differences are likely to occur in the left RGB-domain image (the purplish-red color within the white dashed ellipse does not exist in the original image and is a color difference), while the right YUV-domain image shows the true color of the original image without color differences. The YUV-domain image has fewer color differences compared to the RGB-domain image. For each point in the image, the Y channel determines its brightness, and the UV channels confirm its chroma (hue).

[0060] Step S2: Use the lookup table for image smoothing to perform image smoothing only on the Y-channel image. Image smoothing is a commonly used technique in digital image processing, which is used to reduce image noise, eliminate unnecessary details or small defects, and make the image clearer and smoother. For example, deburring is a typical image smoothing process. For the image smoothing method, any one of mean filtering, median filtering, Gaussian filtering, and bilateral filtering can be adopted. The main purpose of image smoothing: to remove noise (such as salt-and-pepper noise, Gaussian noise, etc.). Smoothing is a basic operation in image processing, and reasonable use can significantly improve the effect of subsequent image analysis and processing. A lookup table is a commonly used data structure in computer science and digital systems, which optimizes computing performance by pre-computing and storing results. The core idea of the lookup table is: pre-compute and store all possible results that may be needed; when a certain computing result is required, directly look up through the index instead of real-time computing. The lookup table is an important technique in performance optimization, especially suitable for scenarios with intensive computing and limited input range. The lookup table significantly improves the computing speed (exchanging space for time) and reduces the processor burden of complex calculations.

[0061] Specifically, the method for obtaining the lookup table for image smoothing includes: As Figure 2 shown in the figure

[0062] S21: Train the first network model for image smoothing. The first input image of the first network model is a non-smooth-edge picture with a size of W*H, and the first label image of the first network model is a smooth-edge picture with a size of W*H. After the training of the first network model converges, convert the mapping relationship between the first input image and the first label image into a rough lookup table for image smoothing. The first input image can be obtained by using non-linear operations to generate certain jagged edges and / or burrs on the image edge. The first label image can be obtained by using a complex neural network to enhance the image edge details.

[0063] The first network model includes: an AI deep neural network (such as CNN, GAN, Transformer); adopting the dense structure in the AI model; the Dense layer is also called the densely connected layer, and multiple densely connected layers are connected to form a densely connected network. The structure of the first network model includes: several convolutional layers and the activation layers corresponding to each convolutional layer. For example, the structure of the first network model includes the first convolutional layer Conv1 to the Nth convolutional layer ConvN, the first to the N-1th convolutional layers correspond to the ReLU activation layer, and the Nth convolutional layer corresponds to the Tanh activation layer. Input the first input image into the first network model, perform model training through several convolutional layers and the activation layers corresponding to each convolutional layer in the first network model, calculate the output image of the first network model and the first label image using the loss function, and when the loss function is minimized, the first network model converges.

[0064] S22. Perform parameter training on the rough look-up table for image smoothing. The first input image for training is an edge-rough image with a size of W*H, and the first label image for training is an edge-smooth image with a size of W*H. After the parameter training converges, the obtained look-up table is the precise look-up table for image smoothing. Use the precise look-up table for image smoothing as the look-up table for image smoothing in step S2. The first input image in step S21 and step S22 is the same, and the first label image in step S21 and step S22 is the same.

[0065] The acquisition of the look-up table for image smoothing and the use of the look-up table for image smoothing to perform image smoothing processing are two independent processes. Obtain the look-up table for image smoothing through network model training in advance, and query the look-up table for image smoothing when image smoothing processing is required.

[0066] In step S2, using the look-up table for image smoothing to perform image smoothing processing specifically includes: as Figure 4 shown, select any initial pixel (such as a5) in the Y-channel image. With the initial pixel as the center, the initial pixel and multiple neighboring pixels surrounding the initial pixel form a first pixel window of m rows * n columns (such as 3 rows * 3 columns or 5 rows * 5 columns). Select at least two first sub-pixel windows in the first pixel window that include the initial pixel (such as a5) and several neighboring pixels with different orientations. Figure 4Four first sub-pixel windows (e.g., Sb, Sc, Sd, and Se) of 2 rows * 2 columns are shown; the first sub-pixel window Sb includes an initial pixel (e.g., a5) and the pixel located in the upper left position of the initial pixel, consisting of a total of 4 pixels; the first sub-pixel window Sc includes an initial pixel (e.g., a5) and the pixel located in the upper right position of the initial pixel, consisting of a total of 4 pixels; the first sub-pixel window Sd includes an initial pixel (e.g., a5) and the pixel located in the lower left position of the initial pixel, consisting of a total of 4 pixels; the first sub-pixel window Se includes an initial pixel (e.g., a5) and the pixel located in the lower right position of the initial pixel, consisting of a total of 4 pixels; all pixel values in each first sub-pixel window are arranged and combined in a certain order as the first pixel index, and the first pixel index starts from the initial pixel (e.g., a5).

[0067] Query the output corresponding to the look-up table for image smoothing, and take the weighted average V of the outputs (e.g., Vb, Vc, Vd, and Ve) corresponding to all first sub-pixel windows 5 As the pixel value of the initial pixel after smoothing; traverse each initial pixel in the Y-channel image, and use the same smoothing method to obtain the Y-channel image after image smoothing processing.

[0068] When storing the image smoothing look-up table, to save memory, sample and store the first pixel index at a specified interval; when the first sub-pixel window queries the image smoothing look-up table, directly query the smoothed pixel value for the first pixel index already stored in the image smoothing look-up table; for the first pixel index not stored in the image smoothing look-up table, interpolate and calculate it by using the stored pixel indices P1, P2...Pn closest to the first pixel index and the corresponding smoothed pixel values V1, V2...Vn; in this way, the memory occupied by the image smoothing look-up table can be reduced.

[0069] Figure 5 It is a schematic diagram before image smoothing for an example image. Figure 6 For Figure 5 It is a schematic diagram after image smoothing for the example image. Figure 7 For Figure 5 The locally enlarged view within the red frame before smoothing intercepted from Figure 6 And the comparison schematic diagram of the locally enlarged view within the red frame after smoothing intercepted from Figure 8 For Figure 5 The locally enlarged view within the blue frame before smoothing intercepted from Figure 6 And the comparison schematic diagram of the locally enlarged view within the blue frame after smoothing intercepted from

[0070] Step S3: Perform image enhancement processing on the Y-channel image after image smoothing using the lookup table for image enhancement. Image enhancement processing is an important technique in digital image processing, aiming to improve the visual effect of the image or highlight key information for subsequent analysis or application. The method for obtaining the lookup table for image enhancement is the same as that for obtaining the lookup table for image smoothing. It can also be referred to Figure 2 As shown, in step S3, the method for obtaining the lookup table for image enhancement includes:

[0071] S31: Train the second network model for image enhancement. The second input image of the second network model is an edge-blurred image with a size of W*H, and the second label image of the second network model is an edge-sharpened image with a size of W*H. After the training of the second network model converges, convert the mapping relationship between the second input image and the second label image into a rough lookup table for image enhancement. The second input image is obtained by destroying the high-frequency details of the image using a weighted calculation kernel with a blur effect. The second label image can be obtained by enhancing the high-frequency details of the image using a weighted calculation kernel with an enhancement effect and a complex neural network.

[0072] The second network model includes: the dense structure in the AI model; the structure of the second network model includes: several convolutional layers and the activation layers corresponding to each convolutional layer. Input the second input image into the second network model, perform model training through several convolutional layers and the activation layers corresponding to each convolutional layer in the second network model, calculate the output image of the second network model and the second label image using the loss function. When the loss function is minimized, the second network model converges.

[0073] S32: Perform parameter training on the rough lookup table for image enhancement. The second input image for training is an edge-blurred image with a size of W*H, and the second label image for training is an edge-sharpened image with a size of W*H. After the training converges, the obtained lookup table is the precise lookup table for image enhancement; use the precise lookup table for image enhancement as the lookup table for image enhancement in step S3. The second input images in step S31 and step S32 are the same, and the second label images in step S31 and step S32 are the same.

[0074] The acquisition of the lookup table for image enhancement and the image enhancement processing using the lookup table for image enhancement are two independent processes. Obtain the lookup table for image enhancement through network model training in advance, and query the lookup table for image enhancement when image enhancement processing is required.

[0075] The image enhancement processing using the lookup table for image enhancement is the same as the image smoothing processing using the lookup table for image smoothing; it can also be referred to Figure 4As shown, in step S3, the image enhancement processing using the lookup table for image enhancement specifically includes: Selecting any pixel to be processed in the Y-channel image after image smoothing processing. Taking the pixel to be processed as the center, a second pixel window of a rows * b columns composed of the pixel to be processed and multiple adjacent pixels surrounding the pixel to be processed is formed; Selecting at least two second sub-pixel windows including the processing pixel and several adjacent pixels with different orientations in the second pixel window; Arranging and combining all pixel values in each second sub-pixel window in a certain order as the second pixel index, querying the corresponding output of the lookup table for image enhancement, and taking the weighted average of the outputs corresponding to all second sub-pixel windows as the pixel value after enhancement of the pixel to be processed; Traversing each pixel to be processed in the Y-channel image after image smoothing processing, and obtaining the Y-channel image after image enhancement processing using the same enhancement method.

[0076] When storing the lookup table for image enhancement, to save memory, the pixel index is sampled and stored at a specified interval. When the second sub-pixel window queries the lookup table for image enhancement, for the second pixel index already stored in the lookup table for image enhancement, the enhanced pixel value is directly queried; For the second pixel index not stored, it is calculated by interpolation using the pixel indices L1, L2... Ln closest to the second pixel index and the corresponding enhanced pixel values Z1, Z2... Zn already stored in the lookup table for image enhancement; In this way, the memory occupied by the lookup table for image enhancement can be reduced. In the interpolation calculation, for example, there are 4 arithmetic expressions including X1*Y1+X2*Y2+X3*Y3+X4*Y4 where any two-digit numbers multiplied pairwise are then added or subtracted. The bit widths of any two digits multiplied pairwise are 8bit and 7bit respectively. The arithmetic expressions for the lookup table interpolation calculation of the present invention do not contain the form of power multiplication, and only contain 4 arithmetic expressions where two numbers are multiplied pairwise and then added or subtracted, which significantly reduces the amount of calculation, does not require the support of special computing power units such as NPU, and both the lookup table and the calculation occupy very little memory.

[0077] Step S4: Perform singular value suppression processing on the Y-channel image after image enhancement processing. Specifically, it includes: Selecting any pixel to be processed in the Y-channel image after the image enhancement processing. Taking the pixel to be processed as the center, a third pixel window of c rows * d columns composed of the pixel to be processed and multiple adjacent pixels surrounding the pixel to be processed is formed; Among the c*d pixel values in the third pixel window, if a certain pixel value is significantly different from other pixel values, then the pixel value is defined as a singular value, and the singular value is corrected to any one of the average value, median value, or weighted average value of the other several pixel values in the third pixel window; If there is no singular value, the original pixel values of the pixels in the third pixel window are maintained. Traversing each pixel to be processed in the Y-channel image after the image enhancement processing, and obtaining the Y-channel image after the image singular value suppression processing using the same singular value suppression processing method.Figure 9 It is the first example diagram for comparing before and after singular value suppression of the Y-channel image after image enhancement processing. Figure 10 It is the second example diagram for comparing before and after singular value suppression of the Y-channel image after image enhancement processing. The singular value suppression processing eliminates the singular pixel points near the black edges in the image, making the image clearer.

[0078] Step S5: Convert the second YUV-domain image composed of the U-channel, V-channel, and the Y-channel image after singular value suppression processing into a high-resolution RGB-domain image to achieve super-resolution processing. Figure 11 It is a comparison diagram of three super-resolution processing effects. Figure 11 In the first column, it is a schematic diagram of directly magnifying the image using a traditional method, such as Bicubic magnification; in the second column, it is a schematic diagram after an existing super-resolution (SR) processing; in the third column, it is a schematic diagram after the super-resolution (SR) processing of the present invention. The present invention obtains a high-resolution RGB-domain image and a clearer image.

[0079] The super-resolution method of the present invention can achieve multi-scale super-resolution, is not limited by fractional magnification; does not rely on special computing units such as NPUs (neural network processors), and ordinary chips can complete the super-resolution function; converts the processing data from the RGB domain to the YUV domain and only processes the Y channel; obtains a lookup table through AI model training, and the size of the lookup table is very small, only 12.8 kBytes for example; improves the AI model by using a dense structure and a more reasonable activation function; improves the accuracy of the AI model through the dense structure and improves the AI model by using a more reasonable activation function. Respectively generate a lookup table for image smoothing and a lookup table for image enhancement, from a rough lookup table to an accurate lookup table; process data in stages to achieve image smoothing and enhancement; remove burrs, supplement details, and enhance clarity through the lookup table. Utilize singular value suppression to eliminate false colors at extremely low cost.

[0080] The present invention provides a method for completing super-resolution technology through a low-cost AI method, which can be implemented in ordinary ISP and DDIC chips without the support of special computing power units such as NPUs. The super-resolution method of the present invention has important application values in many fields such as security monitoring, medical imaging, mobile phone photography, video display, etc.

[0081] The application scenarios of the super-resolution method of the present invention include: for example, film restoration to improve the resolution of old movies and classic videos; medical imaging to enhance the clarity of CT and MRI images for auxiliary diagnosis; satellite / drone images to improve the resolution of remote sensing images; mobile devices to enhance the images after mobile phone photography; games and VR to improve the resolution of game textures in real time and other suitable applications.

[0082] Figure 12It is a comparison chart of various super-resolution processing effects. Figure 12 In the first column, it is a schematic diagram of directly magnifying the image by a traditional method, such as Bicubic magnification; in the second column, it is a schematic diagram after super-resolution (SR) processing by an existing complex neural network method (such as FSRCNN). The FSRCNN network model requires the support of special computing power units with extremely large computing power, such as GPUs and NPUs, and the hardware cost is high. Figure 12 In the third column, it is a schematic diagram after the super-resolution (SR) processing of the present invention. The fourth column is the labeled image. Figure 12 The second row is a partial magnification schematic diagram of the first row. Compared with directly magnifying the image by a traditional method, such as Bicubic magnification (the first column), the present invention (the third column) achieves far better performance on the basis of slightly increasing power consumption, and the obtained image is clearer. Compared with the complex neural network method (the second column), the present invention (the third column) can achieve similar effects without relying on special computing power units such as NPUs. The present invention can realize the iterative update of the solution by updating the look-up table without changing the hardware architecture, and has high flexibility.

[0083] In summary, the present invention provides a super-resolution method, which can achieve multi-scale super-resolution and is not limited by fractional magnification factors; it converts the processed image from the RGB domain to the YUV domain and only processes the Y channel; through AI model training, a look-up table for image smoothing and the look-up table for image enhancement are obtained, and the look-up table occupies very little memory, only about a dozen kilobytes. The look-up table for image smoothing and the look-up table for image enhancement are generated separately in stages, from a rough look-up table to an accurate look-up table; the data is processed in stages to achieve smoothing and enhancement; singular value suppression is used to eliminate false colors at extremely low cost. It does not rely on special computing units such as NPUs (neural network processors), and ordinary chips can complete the super-resolution function. While finally obtaining a high-resolution clear image, the super-resolution method of the present invention does not rely on special computing units, and realizes super-resolution with low power consumption through a look-up table that occupies very little memory, reducing costs.

[0084] In this specification, each embodiment is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the embodiments can be referred to each other. For the method disclosed in the embodiment, since it corresponds to the device disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the method part.

[0085] The above description is only a description of the preferred embodiments of the present invention, and does not limit any scope of the rights of the present invention. Any person skilled in the art can make possible changes and modifications to the technical solution of the present invention by using the methods and technical contents disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and decorations made to the above embodiments according to the technical essence of the present invention without departing from the technical solution of the present invention shall fall within the protection scope of the technical solution of the present invention.

Claims

1. A super-resolution method, characterized in that, Including: Step S1: Provide a low-resolution image, perform preprocessing of resizing and tone mapping on the low-resolution image to obtain a first RGB-domain image corresponding to the width and height of the target high-resolution image, and convert the first RGB-domain image into a first YUV-domain image. The first YUV-domain image includes three-channel images of a Y channel, a U channel, and a V channel; Step S2: Only perform image smoothing processing on the Y-channel image using a lookup table for image smoothing; Step S3: Perform image enhancement processing on the Y-channel image after image smoothing using a lookup table for image enhancement; Step S4: Perform singular value suppression processing on the Y-channel image after image enhancement processing; Step S5: Convert the second YUV-domain image composed of the U channel, the V channel, and the Y-channel image after singular value suppression processing into a high-resolution RGB-domain image; Among them, both the lookup table for image smoothing and the lookup table for image enhancement are obtained through AI model training.

2. The super-resolution method according to claim 1, characterized in that In step S2, the method for obtaining the lookup table for image smoothing includes: S21: Train a first network model for image smoothing. The first input image of the first network model is an edge-unsmoothed picture with a size of W*H, and the first label image of the first network model is an edge-smoothed picture with a size of W*H; after the training of the first network model converges, convert the mapping relationship between the first input image and the first label image into a rough lookup table for image smoothing; S22: Perform parameter training on the rough lookup table for image smoothing. The trained first input image is an edge-unsmoothed picture with a size of W*H, and the trained first label image is an edge-smoothed picture with a size of W*H; after the parameter training converges, the obtained lookup table is an accurate lookup table for image smoothing; use the accurate lookup table for image smoothing as the lookup table for image smoothing in step S2.

3. The super-resolution method according to claim 1, characterized in that In step S3, the method for obtaining the lookup table for image enhancement includes: S31: Train a second network model for image enhancement. The second input image of the second network model is an edge-blurred picture with a size of W*H, and the second label image of the second network model is an edge-sharp picture with a size of W*H; after the training of the second network model converges, convert the mapping relationship between the second input image and the second label image into a rough lookup table for image enhancement; S32: Perform parameter training on the rough lookup table for image enhancement. The trained second input image is an edge-blurred picture with a size of W*H, and the trained second label image is an edge-sharp picture with a size of W*H; after the training converges, the obtained lookup table is the accurate lookup table for image enhancement; use the accurate lookup table for image enhancement as the lookup table for image enhancement in step S3.

4. The super-resolution method according to claim 2 or 3, characterized in that Step S21 further includes: obtaining the first input image by generating certain sawtooth and / or burrs on the image edge using a non-linear operation; Step S31 further includes: obtaining the second input image by destroying the high-frequency details of the image using a weighted calculation kernel with a blur effect.

5. The super-resolution method according to claim 2 or 3, characterized in that In step S21, the first label image is obtained by enhancing the image edge details using a complex neural network; In step S31, the second label image can be obtained by enhancing the high-frequency details of the image using a weighted calculation kernel with an enhancement effect and the complex neural network.

6. The super-resolution method according to claim 2 or 3, characterized in that Both the first network model and the second network model include: a dense structure in the AI model; the structures of both the first network model and the second network model include: several convolutional layers and the activation layers corresponding to the convolutional layers respectively.

7. The super-resolution method according to claim 1, characterized in that Step S2 specifically includes: selecting any initial pixel in the Y-channel image, using the initial pixel as the center, and forming a first pixel window of m rows * n columns composed of the initial pixel and a plurality of neighboring pixels around the initial pixel; selecting at least two first sub-pixel windows including the initial pixel and several of the neighboring pixels with different orientations in the first pixel window; arranging all the pixel values in each of the first sub-pixel windows in a certain order and combining them as a first pixel index, querying the output corresponding to the look-up table of image smoothing, and taking the weighted average of the outputs corresponding to all the first sub-pixel windows as the pixel value of the initial pixel after smoothing; traversing each initial pixel in the Y-channel image and using the same smoothing method to obtain the Y-channel image after image smoothing processing.

8. The super-resolution method according to claim 7, characterized in that When storing the look-up table of image smoothing, sampling and storing the first pixel index at a specified interval; when the first sub-pixel window queries the look-up table of image smoothing, directly querying the smoothed pixel value for the first pixel index already stored in the look-up table of image smoothing; for the first pixel index not stored in the look-up table of image smoothing, interpolating and calculating it by using the pixel indices P1, P2... Pn closest to the first pixel index and the corresponding smoothed pixel values V1, V2... Vn already stored.

9. The super-resolution method according to claim 1, characterized in that Step S3 specifically includes: selecting any pixel to be processed in the Y-channel image after the image smoothing process, using the pixel to be processed as the center, and forming a second pixel window of a rows * b columns composed of the pixel to be processed and multiple neighboring pixels surrounding the pixel to be processed; selecting at least two second sub-pixel windows in the second pixel window, which include the pixel to be processed and several neighboring pixels with different orientations; arranging and combining all the pixel values in each second sub-pixel window in a certain order as the second pixel index, querying the corresponding output of the look-up table for image enhancement, and taking the weighted average of the outputs corresponding to all the second sub-pixel windows as the pixel value of the pixel to be processed after enhancement; traversing each pixel to be processed in the Y-channel image after the image smoothing process, and obtaining the Y-channel image after the image enhancement process by using the same enhancement method. When storing the look-up table for image enhancement, sampling and storing the second pixel index at a specified interval; when the second sub-pixel window queries the look-up table for image enhancement, directly querying the enhanced pixel value for the second pixel index that has been stored in the look-up table for image enhancement; for the second pixel index that has not been stored, interpolating and calculating it by using the pixel indices L1, L2... Ln that are closest to the second pixel index and the corresponding enhanced pixel values Z1, Z2... Zn that have been stored in the look-up table for image enhancement.

10. The super-resolution method according to claim 1, wherein In step S4, the singular value suppression process for the Y-channel image after the image enhancement process specifically includes: selecting any pixel to be processed in the Y-channel image after the image enhancement process, using the pixel to be processed as the center, and forming a third pixel window of c rows * d columns composed of the pixel to be processed and multiple neighboring pixels surrounding the pixel to be processed; among the c * d pixel values in the third pixel window, if a certain pixel value is significantly abnormal compared to other pixel values, then the pixel value is defined as a singular value, and the singular value is corrected to any one of the average value, median value, or weighted average value of the other several pixel values in the third pixel window; if there is no singular value, then keep the original pixel values of the pixels in the third pixel window. Traversing each pixel to be processed in the Y-channel image after the image enhancement process, and obtaining the Y-channel image after the image singular value suppression process by using the same singular value suppression process method.

Images