Image super-resolution processing method and computer device

Abstract

The present invention discloses an image super-resolution processing method and a computer device. The image super-resolution processing method comprises: obtaining an image to be processed, determining an enlarged image corresponding to the image to be processed according to the image to be processed and a preset super-resolution multiple; for each target pixel in the enlarged image, determining a target gradient corresponding to the target pixel based on the target pixel value and the super-resolution multiple of the target pixel; determining a super-resolution pixel value corresponding to the target pixel based on the target gradient, the super-resolution multiple and the image to be processed; adjusting the enlarged image according to the super-resolution pixel values ​​corresponding to each target pixel, and obtaining a super-resolution image corresponding to the image to be processed. In the present invention, only simple operations are needed to determine the target gradient corresponding to the target pixel and the super-resolution pixel value corresponding to the target pixel. The computing power required by the method is much less than that required by the deep learning algorithm, and can be applied to products with low computing power to achieve real-time video super-resolution in products with low computing power.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image super-resolution processing method and a computer device. Background Art

[0002] Image super-resolution technology refers to the process of generating high-resolution images based on low-resolution images. Video super-resolution technology is to generate high-resolution image frames based on each image frame in the video to obtain a high-resolution video. Real-time video super-resolution technology can render low-resolution videos into high-resolution videos when the display device plays the video.

[0003] At present, neural network acceleration chips can be used to quickly obtain super-resolution images corresponding to low-resolution images, and thus, obtain real-time video super-resolution effects with better results. However, the cost of this chip is high and it can only be used in high-end products. If the deep learning-based algorithm in this chip requires high computing power, the deep learning-based algorithm cannot quickly obtain super-resolution images in low- and mid-end products, and thus the real-time video super-resolution task cannot be completed.

[0004] Therefore, the prior art needs to be further improved. Summary of the invention

[0005] The present invention provides an image super-resolution processing method and a computer device, so as to realize rapid generation of super-resolution images in low- and medium-end products, and further realize real-time video super-resolution in low- and medium-end products.

[0006] In a first aspect, an embodiment of the present invention provides an image super-resolution processing method, comprising:

[0007] Acquire an image to be processed, and determine an enlarged image corresponding to the image to be processed according to the image to be processed and a preset super-resolution multiple;

[0008] For each target pixel in the enlarged image, determine a target gradient corresponding to the target pixel based on the target pixel value of the target pixel and the super-resolution multiple, and determine a super-resolution pixel value corresponding to the target pixel based on the target gradient, the super-resolution multiple and the image to be processed;

[0009] The enlarged image is adjusted according to the super-resolution pixel values ​​corresponding to each target pixel point to obtain a super-resolution image corresponding to the image to be processed.

[0010] In one implementation, determining the enlarged image corresponding to the image to be processed according to the image to be processed and a preset super-resolution multiple specifically includes:

[0011] Determine a number of target pixel points according to the image to be processed and the super-resolution multiple;

[0012] For each target pixel point, determining a target pixel value of the target pixel point according to the image to be processed;

[0013] An enlarged image is determined according to a plurality of target pixel points and target pixel values ​​respectively corresponding to the plurality of target pixel points.

[0014] In one implementation, for each target pixel point, determining a target pixel value of the target pixel point according to the image to be processed specifically includes:

[0015] For each target pixel, determine a first position and a plurality of second positions corresponding to the target pixel in the image to be processed, wherein the coordinate of the target pixel is the product of the coordinate of the first position and the super-resolution multiple, and the displacement between the coordinate of each second position and the coordinate of the first position is equal;

[0016] Obtaining a first pixel value corresponding to the first position and second pixel values ​​corresponding to each of the plurality of second positions;

[0017] An average pixel value of the first pixel value and each second pixel value is determined, and the average pixel value is used as the target pixel value.

[0018] In one implementation, determining the first position and the plurality of second positions corresponding to the target pixel point in the image to be processed specifically includes:

[0019] Determining a first position corresponding to the target pixel in the image to be processed;

[0020] The plurality of second positions are determined in the image to be processed according to the first position and a preset expansion value, wherein a displacement between each second position and the first position is equal to the expansion value.

[0021] In one implementation, for each target pixel in the enlarged image, determining a target gradient corresponding to the target pixel based on a target pixel value of the target pixel and the super-resolution multiple specifically includes:

[0022] For each target pixel point in the enlarged image, obtaining a number of reference pixel values ​​corresponding to the target pixel value;

[0023] Determine an initial gradient corresponding to the target pixel value according to the target pixel value and the plurality of reference pixel values;

[0024] If the initial gradient is less than 0, the initial gradient is corrected according to the super-resolution factor to obtain a target gradient.

[0025] In one implementation, the image to be processed is in YUV format, the target pixel value includes a target Y component value, each reference pixel value includes a reference Y component value, and the initial gradient includes an initial gradient of the Y component in a row direction and an initial gradient of the Y component in a column direction; determining the initial gradient corresponding to the target pixel value according to the target pixel value and the plurality of reference pixel values ​​specifically includes:

[0026] An initial Y component gradient in the row direction and an initial Y component gradient in the column direction corresponding to the target pixel value are determined according to the target Y component value and a plurality of reference Y component values.

[0027] In one implementation, the target gradient includes a Y component target gradient in a row direction; if the initial gradient is less than 0, the initial gradient is corrected according to the super-resolution multiple to obtain the target gradient, specifically including:

[0028] If the initial gradient of the Y component in the row direction is less than 0, determining the gradient direction of the initial gradient of the Y component in the row direction according to a number of reference Y component values ​​to obtain an intermediate gradient of the Y component in the row direction;

[0029] The Y component intermediate gradient in the row direction is normalized, and the normalized result is corrected according to the super-resolution multiple to obtain the Y component target gradient in the row direction.

[0030] In one implementation, the normalizing the Y component intermediate gradient in the row direction and correcting the normalized result according to the super-resolution multiple to obtain the Y component target gradient in the row direction specifically includes:

[0031] Normalizing the Y component intermediate gradient in the row direction to obtain a Y component normalized gradient in the row direction;

[0032] Determine a correction coefficient according to a preset edge strength coefficient and the super-resolution multiple;

[0033] The product of the normalized Y-component gradient in the row direction and the correction coefficient is calculated to obtain a target Y-component gradient in the row direction.

[0034] In one implementation, the target gradient includes a Y component target gradient in a column direction; if the initial gradient is less than 0, the initial gradient is corrected according to the super-resolution multiple to obtain the target gradient, specifically including:

[0035] If the initial gradient of the Y component in the column direction is less than 0, determining the gradient direction of the initial gradient of the Y component in the column direction according to the reference Y component values ​​to obtain an intermediate gradient of the Y component in the column direction;

[0036] The Y component intermediate gradient in the column direction is normalized, and the normalized result is corrected according to the super-resolution multiple to obtain the Y component target gradient in the column direction.

[0037] In one implementation, the normalizing the intermediate gradient of the Y component in the column direction and correcting the normalized result according to the super-resolution multiple to obtain the target gradient of the Y component in the column direction specifically includes:

[0038] Normalizing the Y component intermediate gradient in the column direction to obtain a Y component normalized gradient in the column direction;

[0039] According to the normalized gradient of the Y component in the column direction and the correction coefficient, a target gradient of the Y component in the column direction is obtained.

[0040] In one implementation, adjusting the enlarged image according to the super-resolution pixel values ​​corresponding to each target pixel point to obtain the super-resolution image corresponding to the image to be processed specifically includes:

[0041] For each target pixel point, the target pixel value corresponding to the target pixel point is replaced according to the super-resolution value corresponding to the target pixel point, so as to obtain a super-resolution image corresponding to the image to be processed.

[0042] In one implementation, the image to be processed is a two-dimensional animation image.

[0043] In a second aspect, an embodiment of the present invention provides an image super-resolution processing device, comprising:

[0044] An enlarged image determining unit, used to obtain an image to be processed, and determine an enlarged image corresponding to the image to be processed according to the image to be processed and a preset super-resolution multiple;

[0045] a super-resolution pixel value determining unit, configured to determine, for each target pixel in the enlarged image, a target gradient corresponding to the target pixel based on the target pixel value of the target pixel and the super-resolution multiple, and determine the super-resolution pixel value corresponding to the target pixel based on the target gradient, the super-resolution multiple and the image to be processed;

[0046] The super-resolution image generating unit is used to replace the target pixel value corresponding to the target pixel point with the super-resolution pixel value corresponding to the target pixel point, so as to obtain the super-resolution image corresponding to the image to be processed.

[0047] In a third aspect, an embodiment of the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0048] Acquire an image to be processed, and determine an enlarged image corresponding to the image to be processed according to the image to be processed and a preset super-resolution multiple;

[0049] For each target pixel in the enlarged image, determine a target gradient corresponding to the target pixel based on the target pixel value of the target pixel and the super-resolution multiple, and determine a super-resolution pixel value corresponding to the target pixel based on the target gradient, the super-resolution multiple and the image to be processed;

[0050] The enlarged image is adjusted according to the super-resolution pixel values ​​corresponding to each target pixel point to obtain a super-resolution image corresponding to the image to be processed.

[0051] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0052] Acquire an image to be processed, and determine an enlarged image corresponding to the image to be processed according to the image to be processed and a preset super-resolution multiple;

[0053] For each target pixel in the enlarged image, determine a target gradient corresponding to the target pixel based on the target pixel value of the target pixel and the super-resolution multiple, and determine a super-resolution pixel value corresponding to the target pixel based on the target gradient, the super-resolution multiple and the image to be processed;

[0054] The enlarged image is adjusted according to the super-resolution pixel values ​​corresponding to each target pixel point to obtain a super-resolution image corresponding to the image to be processed.

[0055] Compared with the prior art, the embodiments of the present invention have the following advantages:

[0056] In an embodiment of the present invention, an image to be processed is obtained, and an enlarged image corresponding to the image to be processed is determined according to the image to be processed and a preset super-resolution multiple; for each target pixel in the enlarged image, a target gradient corresponding to the target pixel is determined based on the target pixel value of the target pixel and the super-resolution multiple, and a super-resolution pixel value corresponding to the target pixel is determined based on the target gradient, the super-resolution multiple and the image to be processed; the enlarged image is adjusted according to the super-resolution pixel values ​​corresponding to each target pixel, so as to obtain a super-resolution image corresponding to the image to be processed. In the present invention, only simple operations are needed to determine the target gradient corresponding to the target pixel and the super-resolution pixel value corresponding to the target pixel. That is to say, the present invention performs super-resolution processing on the image to be processed, and the processing process of obtaining the super-resolution image does not involve complex operations. The computing power required by the method is much less than the computing power required by the deep learning algorithm. The method can be applied to products with low computing power. In the present invention, after calculating the target gradient corresponding to each pixel to be processed, the target pixel is obtained according to the super-resolution multiple, and then the target pixel is corrected according to the target gradient to obtain a super-resolution image. In the process of generating the super-resolution image, each initial pixel in the enlarged image is traversed to determine the initial pixel value of each initial pixel to obtain the enlarged image, and each initial pixel in the enlarged image is traversed again to determine the super-resolution pixel value of each target pixel. The whole process only needs to be traversed twice, so that the super-resolution image can be quickly generated, and real-time video super-resolution can be realized in products with low computing power. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0058] Figure 1 A schematic diagram of a process flow of an image super-resolution processing method according to an embodiment of the present invention;

[0059] Figure 2 is a schematic diagram of an image super-resolution processing device according to an embodiment of the present invention;

[0060] Figure 3 The figure is a diagram showing the internal structure of a computer device for image enhancement in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] After research, the inventors found that image super-resolution technology refers to the process of generating high-resolution images based on low-resolution images, video super-resolution technology is to generate high-resolution image frames based on each image frame in the video to obtain a high-resolution video, and real-time video super-resolution technology can render low-resolution videos into high-resolution videos when the display device plays the video.

[0063] At present, hardware manufacturers such as Qualcomm, MediaTek, Samsung, and HiSilicon have proposed neural network acceleration chips for algorithms based on deep learning. The advantage is that independent algorithm chips have strong computing power, and the effect of deep learning algorithms is generally better than that of traditional image processing algorithms. However, such chips are expensive and are basically only used in high-end products. Although such chips currently guarantee real-time performance in low-resolution tasks such as faces and gestures, there is still a computing power bottleneck in super-resolution tasks, and they are mostly used for non-real-time tasks such as image super-resolution. Deep learning algorithms cannot quickly obtain super-resolution images, and thus cannot complete real-time video super-resolution tasks.

[0064] In order to solve the above problems, in an embodiment of the present invention, an image to be processed is obtained, and an enlarged image corresponding to the image to be processed is determined according to the image to be processed and a preset super-resolution multiple; for each target pixel in the enlarged image, a target gradient corresponding to the target pixel is determined based on the target pixel value of the target pixel and the super-resolution multiple, and a super-resolution pixel value corresponding to the target pixel is determined based on the target gradient, the super-resolution multiple and the image to be processed; the enlarged image is adjusted according to the super-resolution pixel values ​​corresponding to each target pixel, so as to obtain a super-resolution image corresponding to the image to be processed. In the present invention, only simple operations are needed to determine the target gradient corresponding to the target pixel and the super-resolution pixel value corresponding to the target pixel. That is to say, the present invention performs super-resolution processing on the image to be processed, and the processing process of obtaining the super-resolution image does not involve complex operations. The computing power required by the method is much less than the computing power required by the deep learning algorithm. The method can be applied to products with low computing power. In the present invention, after calculating the target gradient corresponding to each pixel to be processed, the target pixel is obtained according to the super-resolution multiple, and then the target pixel is corrected according to the target gradient to obtain a super-resolution image. In the process of generating the super-resolution image, each initial pixel in the enlarged image is traversed to determine the initial pixel value of each initial pixel to obtain the enlarged image, and each initial pixel in the enlarged image is traversed again to determine the super-resolution pixel value of each target pixel. The whole process only needs to be traversed twice, so that the super-resolution image can be quickly generated, and real-time video super-resolution can be realized in products with low computing power.

[0065] An image super-resolution processing method proposed in an embodiment of the present invention can be applied to electronic devices, such as PCs, servers, mobile phones, televisions, etc. In addition, the functions implemented by the method can be implemented by calling application code through processing in the electronic device. Of course, the program code can be stored in a computer storage medium. It can be seen that the electronic device at least includes a processor and a storage medium.

[0066] See also Figure 1 , Figure 1 An image super-resolution processing method in an embodiment of the present invention is shown. In this embodiment, the image super-resolution processing method may include the following steps:

[0067] S1. Acquire an image to be processed, and determine an enlarged image corresponding to the image to be processed according to the image to be processed and a preset super-resolution multiple.

[0068] In an embodiment of the present invention, the image to be processed is an image frame in a low-resolution video, the low-resolution video may be a video currently being played by an electronic device, and the image to be processed may be an image frame in the low-resolution video that is about to be played. For example, the frames in the low-resolution video are sorted according to the playing time of the low-resolution video, and when the 100th image frame is played, the 101st image frame is obtained, and the 101st image frame is the image to be processed; the low-resolution video may be a local video in an electronic device, and the image to be processed may be any image frame in the low-resolution video.

[0069] In an embodiment of the present invention, the resolution of the image to be processed is smaller than the resolution of the enlarged image, and the resolution of the enlarged image is equal to the resolution of the image to be processed multiplied by the super-resolution multiple. The enlarged image can be determined according to the image to be processed and the super-resolution multiple.

[0070] Specifically, step S1 includes:

[0071] S11. Determine a number of target pixels according to the image to be processed and a preset super-resolution multiple.

[0072] In an embodiment of the present invention, the target pixels are pixels in the enlarged image. The so-called determining the target pixels refers to determining the coordinates corresponding to the target pixels, determining the resolution of the enlarged image according to the resolution of the image to be processed and the super-resolution multiple, and then determining the target pixels according to the resolution of the enlarged image. The image to be processed includes a number of pixels to be processed, and the number of pixels to be processed in the image to be processed multiplied by the preset super-resolution multiple is equal to the number of target pixels in the enlarged image. The resolution of the image to be processed is h×w, the preset super-resolution multiple is 2, and the resolution of the enlarged image is H×W, where H is equal to twice h, that is, H=2h, and similarly, W=2w.

[0073] For example, if the resolution of the image to be processed is 100*100 and the preset super-resolution factor is 2, the resolution of the enlarged image is 200*200. It can be seen that the target pixel point in the enlarged image is (x, y), where the value range of x is [1,200] and the value range of y is [1,200].

[0074] S12: For each target pixel point, determine a target pixel value of the target pixel point according to the image to be processed.

[0075] In an embodiment of the present invention, the image to be processed includes a plurality of pixels to be processed, the pixel value of each pixel to be processed is known, and the target pixel value is determined according to the pixel value of each pixel to be processed. Specifically, the pixel to be processed corresponding to the target pixel is first determined, and the pixel value of the target pixel is determined according to the pixel values ​​of the pixel to be processed and the pixel values ​​of the neighboring pixels of the pixel to be processed.

[0076] Step S12 includes:

[0077] S121. For each target pixel point, determine a first position and several second positions corresponding to the target pixel point in the image to be processed, wherein the coordinates of the first position multiplied by the super-resolution multiple are equal to the coordinates of the target pixel point, and the displacement between the coordinates of each second position and the coordinates of the first position are equal.

[0078] In an embodiment of the present invention, the horizontal coordinate and the vertical coordinate of the target pixel are obtained, and the coordinate of the target pixel is the product of the coordinate of the first position and the super-resolution multiple, and the vertical coordinate of the first position multiplied by the super-resolution multiple is equal to the vertical coordinate of the target pixel. For example, the preset super-resolution multiple is recorded as: s, when the coordinate of the target pixel is (x, y), the coordinate of the pixel to be processed is (x / s, y / s).

[0079] In the embodiment of the present invention, the displacement between the coordinates of each second position and the coordinates of the first position is equal, that is, the plurality of second positions are all located around the first position.

[0080] Specifically, step S121 includes:

[0081] S1211, determining a first position corresponding to the target pixel in the image to be processed;

[0082] S1212. Determine the plurality of second positions according to the first position and a preset expansion value, wherein a displacement between each second position and the first position is equal to the expansion value.

[0083] In an embodiment of the present invention, the plurality of second positions surround the first position, and the distance between each of the plurality of second positions and the first position is equal to the expansion value. The plurality of second positions may be 8, that is, 8 second positions are determined based on the first position and the expansion value with the first position as the center.

[0084] In order to reduce the amount of data, there are four second positions, each of which is in the horizontal direction or vertical direction of the first position, that is, the second positions are respectively on the horizontal left side, the horizontal right side, the vertical upper side and the vertical lower side of the first position. Since there are only four second positions, the amount of data is reduced, thereby improving the image super-resolution processing speed. The dilation value (DILATION, d) is an artificially set empirical parameter, and the dilation value can be 0.5, or the dilation value can be 1.

[0085] For example, if the coordinates of the target pixel are (x, y), then the coordinates of the pixel to be processed are (x / s, y / s), and the coordinates of several neighboring pixels are: and

[0086] S122. Obtain a first pixel value corresponding to the first position and second pixel values ​​corresponding to each of the plurality of second positions, calculate an average pixel value based on the first pixel value and each second pixel value, and use the average pixel value as the target pixel value.

[0087] In an embodiment of the present invention, if the coordinates of the first position or the second position are integers, the pixel value of the pixel to be processed corresponding to the first position is directly obtained in the image to be processed, or the pixel value of the pixel to be processed corresponding to the second position is obtained. The horizontal coordinate or the vertical coordinate of the first position may not be an integer. Therefore, the pixel to be processed corresponding to the first position cannot be found in the image to be processed. The pixel value corresponding to the first position can be determined based on the neighborhood pixels of the first position. Specifically, the pixel value corresponding to the first position can be determined by a bilinear sampling method, or when the coordinates corresponding to the second position are not integers, the pixel value corresponding to the second position can be determined by a bilinear sampling method.

[0088] For example, when the coordinates of the first position are (2, 2.5), the first pixel value of the pixel to be processed with coordinates (2, 3) and the second pixel value of the pixel to be processed with coordinates (2, 2) in the image to be processed are obtained, and the average of the first pixel value and the second pixel value is calculated to obtain the pixel value corresponding to the first position with coordinates (2, 2.5).

[0089] In an embodiment of the present invention, the average value of each candidate pixel value and the pixel value to be processed is calculated to obtain the target pixel value. For example, the coordinates of the target pixel point are P0 = (8, 8), the preset super-resolution multiple is 4, and the preset expansion value is 0.5, then the coordinates of the pixel point to be processed are p0 = (2, 2), then the coordinates of several neighboring pixel points are: p1 = (1.5, 2), p2 = (2.5, 2), p3 = (2, 2.5) and p4 = (2, 1.5), respectively, and the pixel values ​​of p0, p1, p2, p3 and p4 are obtained respectively, and the pixel value of the target pixel point P0 is obtained according to the pixel values ​​of p0, p1, p2, p3 and p4.

[0090] The process of step S12 is implemented according to formula (1).

[0091]

[0092] Among them, Sr is the reciprocal of s, that is, Sr = 1 / s, is the target pixel value of the target pixel point with coordinates (i, j) in the enlarged image, is the pixel value to be processed at the pixel point to be processed with coordinates (i×Sr, j×Sr) in the image to be processed, is the pixel value to be processed at the pixel point to be processed with coordinates (i×Sr-d,j×Sr) in the image to be processed, is the pixel value to be processed at the pixel point to be processed with coordinates (i×Sr+d,j×Sr) in the image to be processed is the pixel value to be processed at the pixel point to be processed with coordinates (i×Sr, j×Sr-d) in the image to be processed, is the pixel value to be processed of the pixel point to be processed with coordinates (i×Sr, j×Sr+d) in the image to be processed.

[0093] Next, the derivation process of formula (1) is introduced.

[0094] The process of determining the enlarged image corresponding to the image to be processed is as follows: determining the initial image corresponding to the image to be processed by interpolation operation, the initial image being an image obtained by directly enlarging the enlarged image by a super-resolution multiple; after obtaining the initial image, filtering the initial image to obtain the enlarged image.

[0095] It includes a plurality of initial pixel points, the number of the initial pixel points being equal to the number of all the pixels to be processed in the image to be processed multiplied by the super-resolution multiple; the pixel value of each initial pixel point of the initial image can be determined according to the pixel value of the pixel point to be processed in the image to be processed. See formula (2).

[0096]

[0097] in, It is the pixel value of the initial pixel with coordinates (i, j) in the initial image, x=i×Sr, y=j×Sr, Sr is the reciprocal of s, that is, Sr=1 / s.

[0098] In an embodiment of the present invention, after obtaining the initial image, the initial image is filtered to obtain the enlarged image. In the prior art, the initial image can be filtered by a convolution layer with a convolution kernel size of 3*3 to obtain the enlarged image. Specifically, the pixel value of the target pixel point with coordinates (x, y) in the enlarged image is determined by 9 initial pixel points in the initial image, and the coordinates of the 9 initial pixel points are: (x-1, y-1), (x, y-1), (x+1, y-1), (x-1, y), (x, y), (x+1, y), (x-1, y+1), (x, y+1), (x+1, y+1); in order to reduce the amount of calculation, the present invention only uses 5 initial pixel points to determine the pixel value of the target pixel point with coordinates (x, y) in the enlarged image, that is, the pixel points with coordinates (x, y-1), (x-1, y), (x, y), (x+1, y) and (x, y+1) are taken.

[0099] Furthermore, the offset is calculated by the dilation value d and the super-resolution, as shown in formula (3).

[0100] D=d×s (3)

[0101] Among them, D is the offset, d is the expansion value, and s is the super-resolution multiple.

[0102] The pixel value of each target pixel in the enlarged image is calculated based on the offset and the initial image, as shown in formula (4).

[0103]

[0104] in, is the target pixel value of the target pixel point with coordinates (i, j) in the enlarged image, is the pixel value of the initial pixel with coordinates (i, j) in the initial image, and D is the offset.

[0105] According to formula (4), when the super-resolution factor is 2 and the dilation value d is 0.5, the offset D is 1, and the target pixel value of the target pixel is determined by the 5 initial pixels in the image block of size 3*3, and the receptive field corresponding to each target pixel is 3; when the super-resolution factor is 4 and the dilation value d is 0.5, the offset D is 2, and the 5 initial pixels include a central pixel with coordinates (i, j) and 4 neighboring pixels around the central pixel. The displacement between each neighboring pixel and the central pixel is 2, that is, the target pixel value of the target pixel is determined by the 5 pixels in the image block of size 5*5, and the receptive field corresponding to each target pixel is 5. It can be seen that the larger the super-resolution factor, the larger the receptive field corresponding to the target pixel. The larger the receptive field corresponding to the target pixel, the larger the area affecting the target pixel, and the more image information included in the target pixel. Therefore, increasing the receptive field can improve the accuracy of subsequent gradient calculation.

[0106] In the embodiment of the present invention, Substituting the expression Xlq in formula (2), we can get formula (2). That is, formula (2) includes: using only 5 initial pixels to determine the pixel of the target pixel with coordinates (x, y) in the enlarged image, which improves the calculation speed and can obtain the enlarged image more quickly, and adding an offset to the initial image, thereby increasing the receptive field corresponding to the target pixel and improving the quality of the enlarged image.

[0107] S2. For each target pixel in the enlarged image, determine the target gradient corresponding to the target pixel based on the target pixel value of the target pixel and the super-resolution multiple, and determine the super-resolution pixel value corresponding to the target pixel based on the target gradient, the super-resolution multiple and the image to be processed.

[0108] In the embodiment of the present invention, in order to further enhance the edge quality in the enlarged image, the target gradient corresponding to each target pixel is calculated, and the target gradient corresponding to each target pixel is determined according to the target gradient. The image to be processed is a two-dimensional animation image. Since the edges in the two-dimensional animation image are generally black lines, it is only necessary to keep the image edge unchanged and shrink the rest to smooth the edge.

[0109] Specifically, step S2 includes:

[0110] S21. For each target pixel point in the enlarged image, obtain a number of reference pixel values ​​corresponding to the target pixel value;

[0111] S22. Determine an initial gradient corresponding to the target pixel value according to the target pixel value and the plurality of reference pixel values.

[0112] In the embodiment of the present invention, the target pixel value is the pixel value of the target pixel point. First, several reference pixel points corresponding to the target pixel point are determined in the enlarged image, and the several reference pixel points are all the neighborhood pixel points of the target pixel point. In other words, the enlarged image includes multiple target pixel points. For a target pixel point, the neighborhood pixel points of the target pixel point are used as the reference pixel points of the target pixel point (the essence of the reference pixel point is also the target pixel point in the enlarged image). Then, the reference pixel values ​​corresponding to each of the several reference pixel points are obtained.

[0113] In an embodiment of the present invention, there are four reference pixels corresponding to the target pixel, which are respectively located on the left side of the target pixel in the row direction, the right side of the target pixel in the row direction, the upper side of the target pixel in the column direction, and the lower side of the target pixel in the column direction. And the displacement between each reference pixel and the target pixel is equal to the order of the initial gradient. The initial gradient is a second-order gradient, that is, the displacement between each reference pixel and the target pixel is equal to 2. For example, when the coordinate of the target pixel is (i, j), the coordinates of several reference pixels are (i-2, j), (i+2, j), (i, j-2), and (i, j+2).

[0114] In the embodiment of the present invention, the image to be processed is in YUV format, "Y" represents brightness, that is, grayscale value; and "U" and "V" represent chromaticity, which is used to describe the color and saturation of the image. Furthermore, the target pixel value includes a target Y component value, and each reference pixel value includes a reference Y component value. The initial gradient includes an initial Y component gradient in the row direction and an initial Y component gradient in the column direction.

[0115] Obtain a target Y component value corresponding to the target pixel value and a reference Y component value corresponding to each reference pixel value grid. Determine an initial Y component gradient in the row direction and an initial Y component gradient in the column direction corresponding to the target pixel value according to the target Y component value and the reference Y component values.

[0116] According to formula (5) and formula (6), the initial gradient of the Y component in the row direction and the initial gradient of the Y component in the column direction corresponding to the target pixel value can be calculated.

[0117]

[0118]

[0119] in, is the initial gradient of the Y component in the row direction corresponding to the coordinate (i, j), is the reference Y component corresponding to the coordinate (i, j-2), is the reference Y component corresponding to the coordinate (i, j+2); is the Y component value of the target corresponding to the coordinate (i, j); is the initial gradient of the Y component in the column direction corresponding to the coordinate (i, j), is the reference Y component corresponding to the coordinate (i-2, j), It is the reference Y component corresponding to the coordinate (i+2,j).

[0120] S23. If the initial gradient is less than 0, the initial gradient is corrected according to the super-resolution factor to obtain a target gradient.

[0121] In the embodiment of the present invention, the image to be processed is a two-dimensional animation image. Since the edges in the two-dimensional animation image are generally black lines, it is only necessary to keep the image edges unchanged and shrink the rest of the image to smooth the edges, that is, to screen out target pixels whose initial gradient is less than 0. For a target pixel, if the initial gradient corresponding to the target pixel is greater than 0, the initial gradient corresponding to the target pixel is modified to 0; if the initial gradient corresponding to the target pixel is less than 0, the initial gradient corresponding to the target pixel is corrected to obtain the target gradient.

[0122] Specifically, step S23 includes:

[0123] S231. If the initial gradient of the Y component in the row direction is less than 0, determine the gradient direction of the initial gradient of the Y component in the row direction based on a number of reference Y component values ​​to obtain the intermediate gradient of the Y component in the row direction; normalize the intermediate gradient of the Y component in the row direction, and correct the normalized result according to the super-resolution multiple to obtain the target gradient of the Y component in the row direction.

[0124] S232. If the initial gradient of the Y component in the column direction is less than 0, determine the gradient direction of the initial gradient of the Y component in the column direction according to the reference Y component values ​​to obtain the intermediate gradient of the Y component in the column direction; normalize the intermediate gradient of the Y component in the column direction, and correct the normalized result according to the super-resolution multiple to obtain the target gradient of the Y component in the column direction.

[0125] In the embodiment of the present invention, the positive or negative initial gradient does not represent the direction of gradient change. and direction to correct.

[0126] The intermediate gradient includes the intermediate gradient of the Y component in the row direction and the intermediate gradient of the Y component in the column direction. Specifically, when the initial gradient of the Y component in the row direction is less than 0, the intermediate gradient of the Y component can be calculated according to formula (7); when the initial gradient of the Y component in the column direction is less than 0, the intermediate gradient of the Y component can be calculated according to formula (8).

[0127]

[0128]

[0129] in, The Y component of the middle gradient in the row direction of the target pixel with coordinates (i, j), It is the median gradient of the Y component in the column direction of the target pixel with coordinates (i, j). sign(·) is a sign function. When () is greater than 0, sign(·) takes the value of 1, and when () is less than 0, sign(·) takes the value of -1.

[0130] In an embodiment of the present invention, the intermediate gradient of the Y component in the row direction is normalized to obtain the normalized gradient of the Y component in the row direction; a correction coefficient is determined according to a preset edge intensity coefficient and the super-resolution multiple; and the product of the normalized gradient of the Y component in the row direction and the correction coefficient is calculated to obtain the target gradient of the Y component in the row direction.

[0131] The Y component intermediate gradient in the row direction is normalized by using formula (9) and formula (10).

[0132]

[0133]

[0134] The preset edge strength coefficient is a set super parameter, S = strength*s, where strength is the edge strength coefficient, s is the super-resolution multiple, and S is the correction coefficient. The product of the correction coefficient and the normalized gradient of the Y component in the row direction can be calculated to obtain the target gradient of the Y component in the row direction.

[0135] In an embodiment of the present invention, the intermediate gradient of the Y component in the column direction is normalized to obtain the normalized gradient of the Y component in the column direction; and the product of the normalized gradient of the Y component in the column direction and the correction coefficient is calculated to obtain the target gradient of the Y component in the column direction.

[0136] The Y component intermediate gradient in the column direction is normalized by using formula (9) and formula (11).

[0137]

[0138] in, It is the normalized gradient of the Y component in the column direction. By calculating the product of the correction coefficient S and the normalized gradient of the Y component in the column direction, the target gradient of the Y component in the column direction can be obtained.

[0139] In the embodiment of the present invention, edge compensation is implemented according to the target gradient and the super-resolution multiple, and the super-resolution pixel value corresponding to the target pixel point can be obtained by formula (12).

[0140]

[0141] Determine the coordinates in the image to be processed as The pixel value is used as the super-resolution pixel value corresponding to (i, j).

[0142] S3. Adjust the enlarged image according to the super-resolution pixel values ​​corresponding to each target pixel point to obtain a super-resolution image corresponding to the image to be processed.

[0143] In the embodiment of the present invention, for each target pixel point, the target pixel value corresponding to the target pixel point is replaced according to the super-resolution value corresponding to the target pixel point, so as to obtain the super-resolution image corresponding to the image to be processed.

[0144] In an embodiment of the present invention, the target pixel point is a pixel point in the enlarged image. For each target pixel point, the super-resolution pixel value corresponding to the target pixel point is determined through step S3, and the target pixel value corresponding to the target pixel point is replaced with the super-resolution pixel value corresponding to the target pixel point to obtain a super-resolution image.

[0145] For example, for the target pixel point f1, the target pixel point value of f1 is g1, and the super-resolution pixel value corresponding to f1 is G1, and the pixel value of f1 is set to G1. The above operation is performed on each target pixel point in the enlarged image, that is, the target pixel points of the enlarged image include: f1, f2, f3, ... fn, for each target pixel point in f1, f2, f3, ... fn, the pixel value of the target pixel point is set to the super-resolution pixel value corresponding to the target pixel point to obtain a super-resolution image.

[0146] In an embodiment of the present invention, an image to be processed is obtained, and an enlarged image corresponding to the image to be processed is determined according to the image to be processed and a preset super-resolution multiple; for each target pixel in the enlarged image, a target gradient corresponding to the target pixel is determined based on the target pixel value of the target pixel and the super-resolution multiple, and a super-resolution pixel value corresponding to the target pixel is determined based on the target gradient, the super-resolution multiple and the image to be processed; the enlarged image is adjusted according to the super-resolution pixel values ​​corresponding to each target pixel, so as to obtain a super-resolution image corresponding to the image to be processed. In the present invention, only simple operations are needed to determine the target gradient corresponding to the target pixel and the super-resolution pixel value corresponding to the target pixel. That is to say, the present invention performs super-resolution processing on the image to be processed, and the processing process of obtaining the super-resolution image does not involve complex operations. The computing power required by the method is much less than the computing power required by the deep learning algorithm. The method can be applied to products with low computing power. In the present invention, after calculating the target gradient corresponding to each pixel to be processed, the target pixel is obtained according to the super-resolution multiple, and then the target pixel is corrected according to the target gradient to obtain a super-resolution image. In the process of generating the super-resolution image, each initial pixel in the enlarged image is traversed to determine the initial pixel value of each initial pixel to obtain the enlarged image, and each initial pixel in the enlarged image is traversed again to determine the super-resolution pixel value of each target pixel. The whole process only needs to be traversed twice, so that the super-resolution image can be generated quickly, and real-time video super-resolution can be performed in products with low computing power.

[0147] Based on the above-mentioned image super-resolution processing method, the embodiment of the present invention further provides an image super-resolution processing device, see Figure 2 ,include:

[0148] An enlarged image determining unit, used to obtain an image to be processed, and determine an enlarged image corresponding to the image to be processed according to the image to be processed and a preset super-resolution multiple;

[0149] a super-resolution pixel value determining unit, configured to determine, for each target pixel in the enlarged image, a target gradient corresponding to the target pixel based on the target pixel value of the target pixel and the super-resolution multiple, and determine the super-resolution pixel value corresponding to the target pixel based on the target gradient, the super-resolution multiple and the image to be processed;

[0150] The super-resolution image generating unit is used to replace the target pixel value corresponding to the target pixel point with the super-resolution pixel value corresponding to the target pixel point, so as to obtain the super-resolution image corresponding to the image to be processed.

[0151] Based on the above-mentioned image super-resolution processing method, an embodiment of the present invention further provides a computer device, which may be a terminal, and the internal structure is as follows: Figure 3 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image super-resolution processing method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0152] Those skilled in the art will understand that Figure 3 What is shown is only a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0153] An embodiment of the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0154] Acquire an image to be processed, and determine an enlarged image corresponding to the image to be processed according to the image to be processed and a preset super-resolution multiple;

[0155] For each target pixel in the enlarged image, determine a target gradient corresponding to the target pixel based on the target pixel value of the target pixel and the super-resolution multiple, and determine a super-resolution pixel value corresponding to the target pixel based on the target gradient, the super-resolution multiple and the image to be processed;

[0156] The enlarged image is adjusted according to the super-resolution pixel values ​​corresponding to each target pixel point to obtain a super-resolution image corresponding to the image to be processed.

[0157] The embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0158] Acquire an image to be processed, and determine an enlarged image corresponding to the image to be processed according to the image to be processed and a preset super-resolution multiple;

[0159] For each target pixel in the enlarged image, determine a target gradient corresponding to the target pixel based on the target pixel value of the target pixel and the super-resolution multiple, and determine a super-resolution pixel value corresponding to the target pixel based on the target gradient, the super-resolution multiple and the image to be processed;

[0160] The enlarged image is adjusted according to the super-resolution pixel values ​​corresponding to each target pixel point to obtain a super-resolution image corresponding to the image to be processed.

[0161] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0162] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for image super-resolution processing, characterized in that: include: Acquire an image to be processed, and determine an enlarged image corresponding to the image to be processed according to the image to be processed and a preset super-resolution multiple; For each target pixel in the enlarged image, determine a target gradient corresponding to the target pixel based on the target pixel value of the target pixel and the super-resolution multiple, and determine a super-resolution pixel value corresponding to the target pixel based on the target gradient, the super-resolution multiple and the image to be processed; According to the super-resolution pixel values ​​corresponding to each target pixel point, the enlarged image is adjusted to obtain a super-resolution image corresponding to the image to be processed; The step of determining, for each target pixel in the enlarged image, a target gradient corresponding to the target pixel based on the target pixel value of the target pixel and the super-resolution multiple specifically includes: For each target pixel point in the enlarged image, obtaining a number of reference pixel values ​​corresponding to the target pixel value; Determine an initial gradient corresponding to the target pixel value according to the target pixel value and the plurality of reference pixel values; If the initial gradient is less than 0, the initial gradient is corrected according to the super-resolution factor to obtain a target gradient; The target gradient includes a Y component target gradient in the row direction; if the initial gradient is less than 0, the initial gradient is corrected according to the super-resolution multiple to obtain the target gradient, specifically including: If the initial gradient of the Y component in the row direction is less than 0, determining the gradient direction of the initial gradient of the Y component in the row direction according to a number of reference Y component values ​​to obtain an intermediate gradient of the Y component in the row direction; Normalizing the Y component intermediate gradient in the row direction, and correcting the normalized result according to the super-resolution multiple to obtain the Y component target gradient in the row direction; The normalizing process is performed on the intermediate gradient of the Y component in the row direction, and the normalized result is corrected according to the super-resolution multiple to obtain the target gradient of the Y component in the row direction, specifically including: Normalizing the Y component intermediate gradient in the row direction to obtain a Y component normalized gradient in the row direction; Determine a correction coefficient according to a preset edge strength coefficient and the super-resolution multiple; The product of the normalized Y-component gradient in the row direction and the correction coefficient is calculated to obtain a target Y-component gradient in the row direction.

2. The image super-resolution processing method according to claim 1, characterized in that: The step of determining, according to the image to be processed and a preset super-resolution multiple, an enlarged image corresponding to the image to be processed specifically includes: Determine a number of target pixel points according to the image to be processed and the super-resolution multiple; For each target pixel point, determining a target pixel value of the target pixel point according to the image to be processed; An enlarged image is determined according to a plurality of target pixel points and target pixel values ​​respectively corresponding to the plurality of target pixel points.

3. The image super-resolution processing method according to claim 2, characterized in that: The step of determining, for each target pixel point, a target pixel value of the target pixel point according to the image to be processed specifically includes: For each target pixel, determine a first position and a plurality of second positions corresponding to the target pixel in the image to be processed, wherein the coordinate of the target pixel is the product of the coordinate of the first position and the super-resolution multiple, and the displacement between the coordinate of each second position and the coordinate of the first position is equal; Obtaining a first pixel value corresponding to the first position and second pixel values ​​corresponding to each of the plurality of second positions; An average pixel value of the first pixel value and each second pixel value is determined, and the average pixel value is used as the target pixel value.

4. The image super-resolution processing method according to claim 3, characterized in that: The step of determining the first position and the plurality of second positions corresponding to the target pixel point in the image to be processed specifically includes: Determining a first position corresponding to the target pixel point in the image to be processed; The plurality of second positions are determined in the image to be processed according to the first position and a preset expansion value, wherein a displacement between each second position and the first position is equal to the expansion value.

5. The image super-resolution processing method according to claim 1, characterized in that: The image to be processed is in YUV format, the target pixel value includes a target Y component value, each reference pixel value includes a reference Y component value, and the initial gradient includes an initial gradient of the Y component in a row direction and an initial gradient of the Y component in a column direction; The determining the initial gradient corresponding to the target pixel value according to the target pixel value and the plurality of reference pixel values ​​specifically includes: An initial Y component gradient in the row direction and an initial Y component gradient in the column direction corresponding to the target pixel value are determined according to the target Y component value and a plurality of reference Y component values.

6. The image super-resolution processing method according to claim 5, characterized in that: The target gradient includes a Y component target gradient in the column direction; if the initial gradient is less than 0, the initial gradient is corrected according to the super-resolution multiple to obtain the target gradient, specifically including: If the initial gradient of the Y component in the column direction is less than 0, determining the gradient direction of the initial gradient of the Y component in the column direction according to the reference Y component values ​​to obtain an intermediate gradient of the Y component in the column direction; The Y component intermediate gradient in the column direction is normalized, and the normalized result is corrected according to the super-resolution multiple to obtain the Y component target gradient in the column direction.

7. The image super-resolution processing method according to claim 6, characterized in that: The normalizing the intermediate gradient of the Y component in the column direction and correcting the normalized result according to the super-resolution multiple to obtain the target gradient of the Y component in the column direction specifically includes: Normalizing the Y component intermediate gradient in the column direction to obtain a Y component normalized gradient in the column direction; A target gradient of the Y component in the column direction is obtained according to the normalized gradient of the Y component in the column direction and the correction coefficient.

8. The image super-resolution processing method according to claim 1, characterized in that: The step of adjusting the enlarged image according to the super-resolution pixel values ​​respectively corresponding to the target pixel points to obtain the super-resolution image corresponding to the image to be processed specifically includes: For each target pixel point, the target pixel value corresponding to the target pixel point is replaced according to the super-resolution value corresponding to the target pixel point, so as to obtain a super-resolution image corresponding to the image to be processed.

9. The image super-resolution processing method according to any one of claims 1 to 8, wherein the image to be processed is a two-dimensional animation image.

10. An image super-resolution processing device, characterized in that: include: An enlarged image determining unit, used to obtain an image to be processed, and determine an enlarged image corresponding to the image to be processed according to the image to be processed and a preset super-resolution multiple; a super-resolution pixel value determining unit, configured to determine, for each target pixel in the enlarged image, a target gradient corresponding to the target pixel based on the target pixel value of the target pixel and the super-resolution multiple, and determine the super-resolution pixel value corresponding to the target pixel based on the target gradient, the super-resolution multiple and the image to be processed; For each target pixel point in the enlarged image, obtaining a number of reference pixel values ​​corresponding to the target pixel value; Determining an initial gradient corresponding to the target pixel value according to the target pixel value and the plurality of reference pixel values; if the initial gradient is less than 0, correcting the initial gradient according to the super-resolution multiple to obtain a target gradient; The target gradient includes a Y component target gradient in the row direction; if the initial gradient is less than 0, the initial gradient is corrected according to the super-resolution multiple to obtain the target gradient, specifically including: If the initial gradient of the Y component in the row direction is less than 0, determining the gradient direction of the initial gradient of the Y component in the row direction according to a number of reference Y component values ​​to obtain an intermediate gradient of the Y component in the row direction; Normalizing the Y component intermediate gradient in the row direction, and correcting the normalized result according to the super-resolution multiple to obtain the Y component target gradient in the row direction; The normalizing process is performed on the intermediate gradient of the Y component in the row direction, and the normalized result is corrected according to the super-resolution multiple to obtain the target gradient of the Y component in the row direction, specifically including: Normalizing the Y component intermediate gradient in the row direction to obtain a Y component normalized gradient in the row direction; Determine a correction coefficient according to a preset edge strength coefficient and the super-resolution multiple; Calculate the product of the normalized gradient of the Y component in the row direction and the correction coefficient to obtain the target gradient of the Y component in the row direction; The super-resolution image generating unit is used to replace the target pixel value corresponding to the target pixel point with the super-resolution pixel value corresponding to the target pixel point, so as to obtain the super-resolution image corresponding to the image to be processed.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images