An image super-resolution processing method and a computer device

Abstract

The present invention discloses an image super-resolution processing method and computer equipment. The image super-resolution processing method includes: obtaining an image to be processed, determining an enlarged image corresponding to the image to be processed based on the image to be processed and a preset super-resolution multiple; for each initial pixel point in the enlarged image, determining a reference vector set corresponding to the initial pixel point; for each initial pixel point, determining a correction value corresponding to the initial pixel point based on the reference vector set corresponding to the initial pixel point, determining a super-resolution pixel value corresponding to the initial pixel point based on the correction value and the image to be processed, and adjusting the enlarged image based on the super-resolution pixel values corresponding to each initial pixel point to obtain a super-resolution image corresponding to the image to be processed. The present invention does not involve complex calculations, and can quickly generate super-resolution images in products with low computing power. The present invention can be applied to real-time video super-resolution in products with low computing power.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image super-resolution processing method and computer equipment. Background Art

[0002] Image super-resolution technology refers to the process of generating high-resolution images based on low-resolution images. Video super-resolution technology is to generate high-resolution image frames based on each image frame in the video to obtain high-resolution video. Real-time video super-resolution technology can render low-resolution videos into high-resolution videos when playing videos on display devices.

[0003] Currently, neural network acceleration chips can be used to quickly obtain super-resolution images corresponding to low-resolution images, thereby achieving better real-time video super-resolution effects. However, the cost of this chip is high and it can only be used in high-end products. If the deep learning-based algorithm in this chip requires high computing power, the deep learning-based algorithm cannot quickly obtain super-resolution images on products with low computing power, and thus cannot complete the real-time video super-resolution task.

[0004] Therefore, the existing technology needs to be further improved. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide an image super-resolution processing method and a computer device to quickly generate super-resolution images in products with low computing power, and further to realize real-time video super-resolution in products with low computing power.

[0006] In a first aspect, an embodiment of the present invention provides an image super-resolution processing method, comprising:

[0007] Acquire an image to be processed, and determine an enlarged image corresponding to the image to be processed according to the image to be processed and a preset super-resolution factor;

[0008] For each initial pixel point in the enlarged image, determining a reference vector set corresponding to the initial pixel point, wherein the reference vector set is used to reflect the pixel values of each pixel point in a neighborhood corresponding to the initial pixel point in the enlarged image;

[0009] For each initial pixel point, determining a correction value corresponding to the initial pixel point according to a reference vector set corresponding to the initial pixel point, and determining a super-resolution pixel value corresponding to the initial pixel point according to the correction value and the image to be processed;

[0010] The enlarged image is adjusted according to the super-resolution pixel values corresponding to the initial pixel points to obtain a super-resolution image corresponding to the image to be processed.

[0011] In one implementation, determining the reference vector set corresponding to the initial pixel point specifically includes:

[0012] Determine several reference pixel points corresponding to the initial pixel point, and obtain a reference pixel value corresponding to each reference pixel point and an initial pixel value corresponding to the initial pixel point, wherein the displacement between each reference pixel point and the initial pixel point is equal;

[0013] An initial vector set is determined according to the obtained initial pixel value and each reference pixel value, and a reference vector set corresponding to the initial pixel point is determined according to the initial vector set.

[0014] In one implementation, the plurality of reference pixels include a first reference pixel, a second reference pixel, a third reference pixel, and a fourth reference pixel; and determining the plurality of reference pixels corresponding to the initial pixel specifically includes:

[0015] Acquire a first reference pixel point and a second reference pixel point located in a row direction of the initial pixel point;

[0016] Acquire a third reference pixel point and a fourth reference pixel point located in a direction of the initial pixel point column.

[0017] In one implementation, the first reference pixel is located on the left side of the initial pixel, the second reference pixel is located on the right side of the initial pixel, the third reference pixel is located below the initial pixel, and the third reference pixel is located above the initial pixel; the image to be processed is in YUV format, the first reference pixel value is the Y component value corresponding to the first reference pixel; the second reference pixel value is the Y component value corresponding to the second reference pixel; the third reference pixel value is the Y component value corresponding to the third reference pixel, and the fourth reference pixel value is the Y component value corresponding to the fourth reference pixel.

[0018] In one implementation, determining an initial vector set based on the obtained initial pixel value and each reference pixel value, and determining a reference vector set corresponding to the initial pixel point based on the initial vector set, specifically includes:

[0019] Converting the initial pixel value into a central initial vector, converting the first reference pixel value corresponding to the first reference pixel into a first initial vector, converting the second reference pixel value corresponding to the second reference pixel into a second initial vector, converting the third reference pixel value corresponding to the third reference pixel into a third initial vector, and converting the fourth reference pixel value corresponding to the fourth reference pixel into a fourth initial vector, to obtain an initial vector set;

[0020] A first target vector is determined according to the first initial vector and the third initial vector, a second target vector is determined according to the second initial vector and the fourth initial vector, and a third target vector is determined according to the center initial vector to obtain a reference vector set.

[0021] In one implementation, determining the correction value corresponding to the initial pixel point according to the reference vector set corresponding to the initial pixel point specifically includes:

[0022] Calculating a first gradient according to the reference vector set;

[0023] If the first gradient is less than 0, the initial gradient is corrected according to the super-resolution factor to obtain a corrected value.

[0024] In one implementation, calculating the first gradient according to the reference vector set specifically includes:

[0025] Calculating a sum of the first target vector and the second target vector to obtain an intermediate vector;

[0026] A first gradient is calculated using a multiplication-addition function according to the intermediate vector and the third target vector.

[0027] In one implementation, if the first gradient is less than 0, correcting the first gradient according to the super-resolution factor to obtain a correction value specifically includes:

[0028] If the first gradient is less than 0, calling a sign function in the image processor to correct the direction of the first gradient to obtain a second gradient;

[0029] Normalizing the second gradient using a normalization function to obtain a third gradient;

[0030] Determining a correction coefficient according to a preset edge strength coefficient and the preset super-resolution multiple;

[0031] The correction value is obtained according to the third gradient and the correction coefficient.

[0032] In one implementation, the image super-resolution processing method is applied to an image processor, and the dimensions of the first target vector, the second target vector, and the third target vector are all configured to be equal to the word length of the computing unit of the image processor, and the dimensions of the first initial vector, the second initial vector, the third initial vector, the fourth initial vector, and the center initial vector are all configured to be half the length of the word length.

[0033] In one implementation, determining the super-resolution pixel value corresponding to the initial pixel point according to the correction value and the image to be processed specifically includes:

[0034] Determine the super-resolution position corresponding to the initial pixel point according to the correction value and the super-resolution multiple;

[0035] The super-resolved pixel value corresponding to the super-resolved position is determined according to the image to be processed.

[0036] In one implementation, determining the super-resolution position corresponding to the initial pixel point according to the correction value and the super-resolution multiple specifically includes:

[0037] Obtaining the initial horizontal coordinate and initial vertical coordinate of the initial pixel point;

[0038] Calculating a first difference between the initial horizontal coordinate and the correction value, and determining the horizontal coordinate of the super-resolution position according to the first difference and the super-resolution multiple;

[0039] A second difference between the initial ordinate and the correction value is calculated, and the ordinate of the super-resolution position is determined according to the second difference and the super-resolution multiple.

[0040] In one implementation, determining the magnified image corresponding to the image to be processed according to the image to be processed and a preset super-resolution factor specifically includes:

[0041] Determine a number of initial pixel points according to the image to be processed and a preset super-resolution multiple;

[0042] For each initial pixel point, determining an initial pixel value of the initial pixel point according to the image to be processed;

[0043] An enlarged image is determined according to a plurality of initial pixel points and initial pixel values respectively corresponding to the plurality of initial pixel points.

[0044] In one implementation, for each initial pixel point, determining the initial pixel value of the initial pixel point according to the image to be processed specifically includes:

[0045] For each initial pixel point, determining a first position and several second positions corresponding to the initial pixel point in the image to be processed, wherein the coordinates of the initial pixel point are the product of the coordinates of the first position and the super-resolution multiple, and the displacement between the coordinates of each second position and the coordinates of the first position is equal;

[0046] Obtaining a first pixel value corresponding to the first position and second pixel values corresponding to each of the plurality of second positions;

[0047] An average pixel value of the first pixel value and each second pixel value is determined, and the average pixel value is used as the initial pixel value.

[0048] In one implementation, determining the first position and several second positions corresponding to the initial pixel point in the image to be processed specifically includes:

[0049] Determining a first position corresponding to the target pixel in the image to be processed;

[0050] The plurality of second positions are determined in the image to be processed according to the first position and a preset expansion value, wherein a displacement between each second position and the first position is equal to the expansion value.

[0051] In one implementation, adjusting the enlarged image according to the super-resolution pixel values corresponding to the initial pixel points to obtain the super-resolution image corresponding to the image to be processed specifically includes:

[0052] For each initial pixel point, the target pixel value corresponding to the initial pixel point is replaced by the super-resolution value corresponding to the initial pixel point to obtain a super-resolution image corresponding to the image to be processed.

[0053] In a second aspect, an embodiment of the present invention provides an image super-resolution processing device, comprising:

[0054] an enlarged image determining unit, configured to obtain an image to be processed, and determine an enlarged image corresponding to the image to be processed according to the image to be processed and a preset super-resolution factor;

[0055] a reference vector set determining unit, configured to determine, for each initial pixel point in the enlarged image, a reference vector set corresponding to the initial pixel point, wherein the reference vector set is configured to reflect pixel values of each pixel point in a neighborhood corresponding to the initial pixel point in the enlarged image;

[0056] a super-resolution pixel value determining unit, configured to determine, for each initial pixel point, a correction value corresponding to the initial pixel point according to a reference vector set corresponding to the initial pixel point, and determine a super-resolution pixel value corresponding to the initial pixel point according to the correction value and the image to be processed;

[0057] The super-resolution image generating unit is used to adjust the amplified image according to the super-resolution pixel values corresponding to the initial pixel points to obtain the super-resolution image corresponding to the image to be processed.

[0058] In a third aspect, an embodiment of the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0059] Acquire an image to be processed, and determine an enlarged image corresponding to the image to be processed according to the image to be processed and a preset super-resolution factor;

[0060] For each initial pixel point in the enlarged image, determining a reference vector set corresponding to the initial pixel point, wherein the reference vector set is used to reflect the pixel values of each pixel point in a neighborhood corresponding to the initial pixel point in the enlarged image;

[0061] For each initial pixel point, determining a correction value corresponding to the initial pixel point according to a reference vector set corresponding to the initial pixel point, and determining a super-resolution pixel value corresponding to the initial pixel point according to the correction value and the image to be processed;

[0062] The enlarged image is adjusted according to the super-resolution pixel values corresponding to the initial pixel points to obtain a super-resolution image corresponding to the image to be processed.

[0063] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0064] Acquire an image to be processed, and determine an enlarged image corresponding to the image to be processed according to the image to be processed and a preset super-resolution factor;

[0065] For each initial pixel point in the enlarged image, determining a reference vector set corresponding to the initial pixel point, wherein the reference vector set is used to reflect the pixel values of each pixel point in a neighborhood corresponding to the initial pixel point in the enlarged image;

[0066] For each initial pixel point, determining a correction value corresponding to the initial pixel point according to a reference vector set corresponding to the initial pixel point, and determining a super-resolution pixel value corresponding to the initial pixel point according to the correction value and the image to be processed;

[0067] The enlarged image is adjusted according to the super-resolution pixel values corresponding to the initial pixel points to obtain a super-resolution image corresponding to the image to be processed.

[0068] Compared with the prior art, the embodiments of the present invention have the following advantages:

[0069] In an embodiment of the present invention, an image to be processed is obtained, and an enlarged image corresponding to the image to be processed is determined based on the image to be processed and a preset super-resolution factor. For each initial pixel in the enlarged image, a reference vector set corresponding to the initial pixel is determined, wherein the reference vector set is used to reflect the pixel values of each pixel in the neighborhood corresponding to the initial pixel in the enlarged image. For each initial pixel, a correction value corresponding to the initial pixel is determined based on the reference vector set corresponding to the initial pixel, and a super-resolution pixel value corresponding to the initial pixel is determined based on the correction value and the image to be processed. For each initial pixel, a correction value corresponding to the initial pixel is determined based on the reference vector set corresponding to the initial pixel, and a super-resolution pixel value corresponding to the initial pixel is determined based on the correction value and the image to be processed. The enlarged image is adjusted based on the super-resolution pixel values corresponding to each initial pixel to obtain a super-resolution image corresponding to the image to be processed. Existing techniques determine the super-resolution image corresponding to the image to be processed through deep learning. Specifically, deep learning is required to determine the correction value corresponding to each initial pixel in the enlarged image and the super-resolution pixel value corresponding to each initial pixel, which requires high computing power. The present invention does not use deep learning methods to generate super-resolution images. Instead, it determines a reference vector set corresponding to each initial pixel point, uses the reference vector set to determine the correction value corresponding to the initial pixel point, and then determines the super-resolution pixel value corresponding to the initial pixel point. In other words, the process of generating super-resolution images in the present invention does not involve complex calculations, and only simple calculations are required to obtain super-resolution images. The computing power required by the present invention is far less than that required by deep learning methods to generate super-resolution images. Super-resolution images can be quickly generated in products with low computing power. Therefore, the present invention can be applied to real-time video super-resolution in products with low computing power. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0071] Figure 1 Schematic diagram of a process for image super-resolution processing according to an embodiment of the present invention;

[0072] Figure 2 Schematic diagram of the structure of an image super-resolution processing device according to an embodiment of the present invention;

[0073] Figure 3 2 is a diagram showing the internal structure of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0074] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0075] After research, the inventors found that image super-resolution technology refers to the process of generating high-resolution images based on low-resolution images, video super-resolution technology is to generate high-resolution image frames based on each image frame in the video to obtain high-resolution video, and real-time video super-resolution technology can render low-resolution videos into high-resolution videos when playing videos on a display device.

[0076] Currently, hardware manufacturers such as Qualcomm, MediaTek, Samsung, and HiSilicon have introduced neural network acceleration chips for deep learning-based algorithms. These chips offer the advantage of independent algorithm chips with high computing power, and deep learning algorithms generally perform better than traditional image processing algorithms. However, these chips are relatively expensive and are generally only used in high-end products. While these chips currently guarantee real-time performance for low-resolution tasks such as faces and gestures, they face computing bottlenecks in super-resolution tasks, and are primarily used for non-real-time tasks such as image super-resolution. Deep learning algorithms cannot quickly generate super-resolution images, making them incapable of real-time video super-resolution.

[0077] To address the above-mentioned issues, in an embodiment of the present invention, an image to be processed is obtained, and an enlarged image corresponding to the image to be processed is determined based on the image to be processed and a preset super-resolution factor; for each initial pixel in the enlarged image, a reference vector set corresponding to the initial pixel is determined, wherein the reference vector set is used to reflect the pixel values of each pixel in the neighborhood corresponding to the initial pixel in the enlarged image; for each initial pixel, a correction value corresponding to the initial pixel is determined based on the reference vector set corresponding to the initial pixel; the super-resolution pixel value corresponding to the initial pixel is determined based on the correction value and the image to be processed; and the enlarged image is adjusted based on the super-resolution pixel values corresponding to each initial pixel to obtain a super-resolution image corresponding to the image to be processed. Existing technologies use deep learning to determine the super-resolution image corresponding to the image to be processed. Specifically, deep learning is required to determine the correction value corresponding to each initial pixel in the enlarged image and the super-resolution pixel value corresponding to each initial pixel, which requires high computing power. The present invention does not use deep learning methods to generate super-resolution images. Instead, it determines a reference vector set corresponding to each initial pixel point, uses the reference vector set to determine the correction value corresponding to the initial pixel point, and then determines the super-resolution pixel value corresponding to the initial pixel point. In other words, the process of generating super-resolution images in the present invention does not involve complex calculations, and only simple calculations are required to obtain super-resolution images. The computing power required by the present invention is far less than that required by deep learning methods to generate super-resolution images. Super-resolution images can be quickly generated in products with low computing power. Therefore, the present invention can be applied to real-time video super-resolution in products with low computing power.

[0078] An image super-resolution processing method proposed in an embodiment of the present invention can be applied to electronic devices equipped with a graphics processing unit (GPU), such as PCs, servers, mobile phones, and televisions equipped with a GPU. When executing the image super-resolution processing method, the sub-device utilizes the GPU to perform part of the process in the image super-resolution processing method. Furthermore, the functions implemented by the method can be implemented by calling application code, which can of course be stored in a computer storage medium. Therefore, the electronic device includes at least a processor and a storage medium.

[0079] See also Figure 1 , Figure 1 An image super-resolution processing method according to an embodiment of the present invention is shown. In this embodiment, the method may include the following steps:

[0080] S1. Acquire an image to be processed, and determine an enlarged image corresponding to the image to be processed according to the image to be processed and a preset super-resolution factor.

[0081] In an embodiment of the present invention, the image to be processed is an image frame in a low-resolution video, the low-resolution video may be a video currently being played by an electronic device, and the image to be processed may be an image frame in the low-resolution video that is about to be played; an image frame that is about to be played refers to an image frame that is located after the image frame currently being played in the image frame sequence corresponding to the video currently being played. After the image frame currently being played is played, the image frame that is about to be played will be played. For example, the image frames in the low-resolution video are sorted according to the playback time of the low-resolution video, and when the 100th image frame is played, the 101st image frame is obtained, then the 101st image frame is the image to be processed; the low-resolution video may be a local video in the electronic device, and the image to be processed may be any image frame in the low-resolution video.

[0082] In an embodiment of the present invention, the resolution of the image to be processed is smaller than the resolution of the enlarged image, and the resolution of the enlarged image is equal to the resolution of the image to be processed multiplied by the super-resolution factor. The enlarged image can be determined based on the image to be processed and the super-resolution factor.

[0083] Electronic devices contain a central processing unit (CPU) and a graphics processing unit (GPU). The GPU is a microprocessor dedicated to image-related computing. In the prior art, the CPU typically copies the image to be processed to the GPU. The GPU processes the image and then copies the processed image back to the CPU. This double copying process results in a longer processing time for the image to be processed. In embodiments of the present invention, this eliminates the need to copy the image to the GPU and then back to the CPU. Instead, a memory space is allocated within the CPU, which the GPU can directly access to process the image.

[0084] First, the image to be processed is read from the memory of the central processing unit and mapped to the video memory of the image processor to obtain the image to be processed. In specific implementation, the mapping function clEnqueueMapBuffer can be called in the Open Computing Language (OpenCL) to achieve the mapping between the CPU and GPU. In other words, the image to be processed does not need to be copied from the CPU to the GPU, which saves the time spent on copying the image to be processed and improves the speed of processing the image to be processed.

[0085] Specifically, step S1 includes:

[0086] S10. Determine a number of initial pixel points according to the image to be processed and a preset super-resolution factor.

[0087] In an embodiment of the present invention, the several initial pixel points are pixel points in the enlarged image. The so-called determining the several initial pixel points refers to determining the coordinates corresponding to each of the several initial pixel points, determining the resolution of the enlarged image according to the resolution of the image to be processed and the super-resolution multiple, and then determining the several initial pixel points according to the resolution of the enlarged image. The image to be processed includes several pixel points to be processed, and the number of pixel points to be processed in the image to be processed multiplied by the preset super-resolution multiple is equal to the number of initial pixel points in the enlarged image. For example, the resolution of the image to be processed is h×w, the preset super-resolution multiple is 2, and the resolution of the enlarged image is H×W, where H is equal to 2 times h, that is, H=2h, and similarly, W=2w.

[0088] For example, if the resolution of the image to be processed is 100*100 and the preset super-resolution factor is 2, the resolution of the enlarged image is 200*200. It can be seen that the initial pixel point in the enlarged image is (x, y), where the value range of x is [1,200] and the value range of y is [1,200].

[0089] S11 . For each initial pixel point, determine an initial pixel value of the initial pixel point according to the image to be processed.

[0090] In an embodiment of the present invention, the image to be processed includes several pixels to be processed, the pixel value of each pixel to be processed is known, and the initial pixel value of the initial pixel in the enlarged image is determined according to the pixel value of each pixel to be processed.

[0091] In an embodiment of the present invention, for each initial pixel point, a computing unit of the GPU is used to calculate the target pixel value, and the GPU processes all the initial pixel points in parallel to obtain the initial pixel values corresponding to each of the initial pixel points. That is, when the size of the enlarged image is H*W, the number of logical units of the GPU parallel calculation is H*W.

[0092] Specifically, step S11 includes:

[0093] S100. For each initial pixel point, determine a first position and several second positions corresponding to the initial pixel point in the image to be processed, wherein the coordinates of the initial pixel point are the product of the coordinates of the first position and the super-resolution multiple, and the displacement between the coordinates of each second position and the coordinates of the first position is equal.

[0094] In an embodiment of the present invention, the abscissa and ordinate of the initial pixel are obtained, and the abscissa of the first position multiplied by the super-resolution factor equals the abscissa of the initial pixel, and the ordinate of the first position multiplied by the super-resolution factor equals the ordinate of the initial pixel. For example, the preset super-resolution factor is denoted as s, and when the coordinates of the initial pixel are (x, y), the coordinates of the pixel to be processed are (x / s, y / s).

[0095] In the embodiment of the present invention, the displacement between the coordinates of each second position and the coordinates of the first position is equal, that is, the plurality of second positions are all located around the first position.

[0096] Specifically, step S100 includes:

[0097] S110, determining a first position corresponding to the target pixel in the image to be processed;

[0098] S111. Determine the plurality of second positions according to the first position and a preset expansion value, wherein a displacement between each second position and the first position is equal to the expansion value.

[0099] In this embodiment of the present invention, to reduce data volume, there are four second positions, each located horizontally or vertically relative to the first position. That is, the second positions are located horizontally to the left, horizontally to the right, vertically above, and vertically below the first position, respectively. Since there are only four second positions, the data volume is reduced, thereby increasing the image super-resolution processing speed. The dilation value (DILATION, d) is an empirically set parameter and can be 0.5 or 1.

[0100] For example, when the expansion value is 0.5, the coordinates of the initial pixel are (x, y), and the coordinates of the pixel to be processed are (x / s, y / s). Then the coordinates of the second positions are: and

[0101] S101. Obtain a first pixel value corresponding to the first position and second pixel values corresponding to each of the plurality of second positions, calculate an average pixel value based on the first pixel value and each second pixel value, and use the average pixel value as the initial pixel value.

[0102] In an embodiment of the present invention, if the coordinates of the first position or the second position are integers, the pixel value of the pixel to be processed corresponding to the first position is directly obtained in the image to be processed, or the pixel value of the pixel to be processed corresponding to the second position is obtained. The horizontal coordinate or the vertical coordinate of the first position may not be an integer. Therefore, the pixel to be processed corresponding to the first position cannot be found in the image to be processed. The pixel value corresponding to the first position can be determined based on the neighborhood pixel points of the first position. The neighborhood pixel points of the first position refer to the pixel points located around the first position in the image to be processed. In other words, the coordinates of the neighborhood pixel points of the first position are less than a preset distance from the coordinates of the first position. The preset distance can be 1, that is, the coordinates of the neighborhood pixel points of the first position are less than 1 pixel unit from the coordinates of the first position.

[0103] Specifically, the pixel value corresponding to the first position may be determined by a bilinear sampling method, or when the coordinate corresponding to the second position is not an integer, the pixel value corresponding to the second position may be determined by a bilinear sampling method.

[0104] For example, when the coordinates of the first position are (2, 2.5), the first pixel value of the pixel to be processed with coordinates (2, 3) and the second pixel value of the pixel to be processed with coordinates (2, 2) in the image to be processed are obtained, and the average of the first pixel value and the second pixel value is calculated to obtain the pixel value corresponding to the first position with coordinates (2, 2.5).

[0105] In an embodiment of the present invention, the average value of each candidate pixel value and the pixel value to be processed is calculated to obtain the initial pixel value. For example, the coordinates of the initial pixel point are P0 = (8, 8), the preset super-resolution multiple is 4, and the preset expansion value is 0.5. The coordinates of the pixel point to be processed are p0 = (2, 2), and the coordinates of several neighboring pixel points are: p1 = (1.5, 2), p2 = (2.5, 2), p3 = (2, 2.5), and p4 = (2, 1.5). The pixel values of p0, p1, p2, p3, and p4 are obtained respectively, and the pixel value of the initial pixel point P0 is obtained based on the pixel values of p0, p1, p2, p3, and p4.

[0106] According to formula (1), the process of step S12 is implemented.

[0107]

[0108] Among them, Sr is the reciprocal of s, that is, Sr = 1 / s, is the initial pixel value of the initial pixel point with coordinates (i, j) in the magnified image, is the pixel value to be processed at the pixel point to be processed with coordinates (i×Sr, j×Sr) in the image to be processed, is the pixel value to be processed at the pixel point to be processed with coordinates (i×Sr-d, j×Sr) in the image to be processed, is the pixel value to be processed at the pixel point to be processed with coordinates (i×Sr+d,j×Sr) in the image to be processed is the pixel value to be processed at the pixel point to be processed with coordinates (i×Sr, j×Sr-d) in the image to be processed, is the pixel value to be processed of the pixel to be processed with coordinates (i×Sr, j×Sr+d) in the image to be processed.

[0109] Next, we will introduce the derivation process of formula (1).

[0110] The process of determining the enlarged image corresponding to the image to be processed is as follows: determining the initial image corresponding to the image to be processed by interpolation operation, where the initial image is directly obtained by enlarging the enlarged image by a super-resolution multiple; after obtaining the initial image, filtering the initial image to obtain the enlarged image.

[0111] The method includes a plurality of initial pixels, the number of which is equal to the number of all pixels to be processed in the image to be processed multiplied by the super-resolution factor. The pixel value of each initial pixel in the initial image can be determined based on the pixel value of the pixel to be processed in the image to be processed. See formula (2).

[0112]

[0113] in, It is the pixel value of the initial pixel point with coordinates (i, j) in the initial image, x = i × Sr, y = j × Sr, Sr is the reciprocal of s, that is, Sr = 1 / s.

[0114] In an embodiment of the present invention, after obtaining the initial image, the initial image is filtered to obtain the enlarged image. In the prior art, the initial image can be filtered by a convolution layer with a convolution kernel size of 3*3 to obtain the enlarged image. Specifically, the pixel value of the initial pixel point with coordinates (x, y) in the enlarged image is determined by 9 initial pixel points in the initial image, and the coordinates of the 9 initial pixel points are respectively: (x-1, y-1), (x, y-1), (x+1, y-1), (x-1, y), (x, y), (x+1, y), (x-1, y+1), (x, y+1), (x+1, y+1); in order to reduce the amount of calculation, the present invention only uses 5 initial pixel points to determine the pixel value of the initial pixel point with coordinates (x, y) in the enlarged image, that is, the pixel points with coordinates (x, y-1), (x-1, y), (x, y), (x+1, y) and (x, y+1) are taken.

[0115] Furthermore, the offset is calculated by the dilation value d and super-resolution, as shown in formula (3).

[0116] D=d×s (3)

[0117] Where D is the offset, d is the expansion value, and s is the super-resolution multiple.

[0118] The pixel value of each initial pixel point in the enlarged image is calculated based on the offset and the initial image, as shown in formula (4).

[0119]

[0120] in, is the initial pixel value of the initial pixel point with coordinates (i, j) in the magnified image, is the pixel value of the initial pixel with coordinates (i, j) in the initial image, and D is the offset.

[0121] According to formula (4), when the super-resolution factor is 2 and the dilation value d is 0.5, the offset D is 1, and the initial pixel value of the initial pixel is determined by the 5 initial pixels in the 3*3 image block. The receptive field corresponding to each initial pixel is 3. When the super-resolution factor is 4 and the dilation value d is 0.5, the offset D is 2. The 5 initial pixels include a central pixel with coordinates (i, j) and 4 neighboring pixels around the central pixel. The displacement between each neighboring pixel and the central pixel is 2. In other words, the initial pixel value of the initial pixel is determined by the 5 pixels in the 5*5 image block. The receptive field corresponding to each initial pixel is 5. It can be seen that the larger the super-resolution factor, the larger the receptive field corresponding to the initial pixel. The larger the receptive field corresponding to the initial pixel. This means that the area affecting the initial pixel is larger and the initial pixel contains more image information. Therefore, increasing the receptive field can improve image quality.

[0122] In the embodiment of the present invention, the formula (4) Substituting the expression Xlq in formula (2) yields formula (2). In other words, formula (2) includes the following: using only five initial pixels to determine the pixel of the initial pixel with coordinates (x, y) in the enlarged image, which improves the calculation speed and allows the enlarged image to be obtained more quickly; and adding an offset to the initial image, thereby increasing the receptive field corresponding to the initial pixel and improving the quality of the enlarged image.

[0123] S2. For each initial pixel point in the enlarged image, determine a reference vector set corresponding to the initial pixel point, wherein the reference vector set is used to reflect the pixel values of each pixel point in a neighborhood corresponding to the initial pixel point in the enlarged image.

[0124] In an embodiment of the present invention, the initial pixel values corresponding to all the initial pixel points in the enlarged image can be obtained according to step S1. In order to enhance the edge quality in the enlarged image, a reference vector set corresponding to each initial pixel point is calculated. The neighborhood corresponding to the initial pixel point refers to the area formed by the pixel points surrounding the initial pixel point with the initial pixel point as the center. The pixel points in the neighborhood corresponding to the initial pixel point include: the initial pixel point and the pixel points surrounding the initial pixel point, and the distance between any pixel point surrounding the initial pixel point and the initial pixel point is less than a preset distance. The preset distance can be 1, that is, the distance between any pixel point surrounding the initial pixel point and the initial pixel point is less than 1 pixel unit.

[0125] In an embodiment of the present invention, each pixel point in the neighborhood corresponding to the initial pixel point includes: the initial pixel point, and several reference pixel points corresponding to the initial pixel point. After determining the initial pixel point and the several reference pixel points corresponding to the initial pixel point, the pixel values corresponding to the initial pixel point and the several reference pixel points are obtained, and then the reference vector set is determined based on the obtained pixel values.

[0126] Specifically, step S2 includes:

[0127] S20. Determine several reference pixel points corresponding to the initial pixel point, and obtain a reference pixel value corresponding to each reference pixel point and an initial pixel value corresponding to the initial pixel point, wherein the displacement between each reference pixel point and the initial pixel point is equal.

[0128] In an embodiment of the present invention, several reference pixels corresponding to an initial pixel are first determined in the magnified image. These reference pixels are all neighborhood pixels of the initial pixel. In other words, the magnified image includes multiple initial pixels. For each initial pixel, the neighborhood pixels of the initial pixel are used as reference pixels for the initial pixel (the reference pixel is essentially the initial pixel in the magnified image). The reference pixel values corresponding to each of the reference pixels are then obtained, along with the initial pixel value corresponding to the initial pixel.

[0129] Specifically, step S20 includes:

[0130] S200 , obtaining a first reference pixel point and a second reference pixel point located in a row direction of the initial pixel point, and obtaining a third reference pixel point and a fourth reference pixel point located in a column direction of the initial pixel point.

[0131] In an embodiment of the present invention, there are 4 reference pixels corresponding to the initial pixel point, and the reference pixels include a first reference pixel point, a second reference pixel point, a third reference pixel point, and a fourth reference pixel point. The first reference pixel point is located on the left side of the initial pixel point, the second reference pixel point is located on the right side of the initial pixel point, the third reference pixel point is located on the lower side of the initial pixel point, and the fourth reference pixel point is located on the upper side of the initial pixel point. The position shift between each reference pixel point and the initial pixel point can be equal to 2. For example, when the coordinates of the initial pixel point are (i, j), the coordinates of the reference pixel points are (i-2, j), (i+2, j), (i, j-2), and (i, j+2).

[0132] In this embodiment of the present invention, the image to be processed is in YUV format, where "Y" represents brightness, i.e., grayscale value; and "U" and "V" represent chrominance, which are used to describe the color and saturation of the image. Furthermore, the first reference pixel value is the Y component value corresponding to the first reference pixel; the second reference pixel value is the Y component value corresponding to the second reference pixel; the third reference pixel value is the Y component value corresponding to the third reference pixel; and the fourth reference pixel value is the Y component value corresponding to the fourth reference pixel.

[0133] S21. Determine an initial vector set according to the obtained initial pixel value and each reference pixel value, and determine a reference vector set corresponding to the initial pixel point according to the initial vector set.

[0134] In an embodiment of the present invention, the initial vector set is determined based on the initial pixel value and each reference pixel value by a processor that executes the image super-resolution method. The processor that executes the image super-resolution method is a graphics processing unit (GPU).

[0135] Specifically, step S21 includes:

[0136] S210. Convert the initial pixel value into a central initial vector, convert the first reference pixel value corresponding to the first reference pixel point into a first initial vector, convert the second reference pixel value corresponding to the second reference pixel point into a second initial vector, convert the third reference pixel value corresponding to the third reference pixel point into a third initial vector, and convert the fourth reference pixel value corresponding to the fourth reference pixel point into a fourth initial vector to obtain an initial vector set.

[0137] In an embodiment of the present invention, the dimensions of the first initial vector, the second initial vector, the third initial vector, the fourth initial vector and the center initial vector are all configured to be half the length of the word length. The image processor can be a Mali Bifrost GPU, that is, the image super-resolution method can be performed using a Mali Bifrost GPU. The word length of the operation unit of the Mali Bifrost GPU is 32 bits, so the obtained initial pixel value and each reference pixel value are converted into a 16-bit floating point number to obtain the initial vectors corresponding to the initial pixel value and each reference pixel value, respectively, and then obtain an initial vector set (including: a center initial vector, a first initial vector, a second initial vector, a third initial vector and a fourth initial vector).

[0138] S211. Determine a first target vector based on the first initial vector and the third initial vector, determine a second target vector based on the second initial vector and the fourth initial vector, and determine a third target vector based on the center initial vector, with reference to a vector set.

[0139] In an embodiment of the present invention, the reference vector set includes: a first target vector, a second target vector, and a third target vector. The dimensions of the first target vector, the second target vector, and the third target vector are all configured to be equal to the word length of the arithmetic unit of the image processor.

[0140] When a Mali Bifrost GPU is used to execute the image super-resolution method, since the word length of the operation unit of the Mali Bifrost GPU is 32 bits, the dimensions of the first target vector, the second target vector, and the third target vector are all configured to be 32. Specifically, the first reference vector and the third reference vector are concatenated into a vector with a dimension of 32, namely, the first target vector; the second reference vector and the fourth reference vector are concatenated into a vector with a dimension of 32, namely, the second target vector; the central initial vector is copied, and the copied vector is concatenated with the central initial vector to obtain the third target vector. This is shown in Formulas (5), (6), and (7).

[0141]

[0142]

[0143]

[0144] in, is the first target vector, is the Y component value corresponding to the first reference pixel with coordinates (i, j-2), is the Y component value corresponding to the third reference pixel with coordinates (i-2, j); is the second target vector, is the Y component value corresponding to the second reference pixel with coordinates (i, j+2), is the Y component value corresponding to the fourth reference pixel with coordinates (i+2,j), is the third target vector, It is the Y component value corresponding to the initial pixel point with coordinates (i, j).

[0145] S3. For each initial pixel point, determine a correction value corresponding to the initial pixel point according to a reference vector set corresponding to the initial pixel point, and determine a super-resolution pixel value corresponding to the initial pixel point according to the correction value and the image to be processed.

[0146] In the embodiment of the present invention, the image to be processed is a two-dimensional animation image. Since the edges in a two-dimensional animation image are generally black lines, it is only necessary to keep the image edges unchanged and shrink the rest to smooth the edges.

[0147] Specifically, step S3 includes:

[0148] S30. Calculate a first gradient according to the reference vector set.

[0149] In an embodiment of the present invention, the reference vector set includes a first target vector, a second target vector, and a third target vector. The sum of the first target vector and the second target vector is calculated to obtain an intermediate vector. A multiplication-addition function is used to calculate a first gradient based on the intermediate vector and the third target vector. When the image super-resolution processing method is executed by an image processor, the multiplication-addition function in the image processor can be called. This is shown in Formulas (8) and (9).

[0150]

[0151]

[0152] in, is the middle vector, is the first gradient. mad(·) is the multiplication-addition function that can be called by the GPU. The essence of formula (9) is, Calling the multiplication and addition function in the image processor to calculate the first gradient can speed up the operation.

[0153] S31. If the first gradient is less than 0, correct the initial gradient according to the super-resolution factor to obtain a corrected value.

[0154] In an embodiment of the present invention, the image to be processed is a two-dimensional animated image. Since the edges in a two-dimensional animated image are generally black lines, it is only necessary to maintain the image edges while shrinking the remaining portions to smooth the edges. In other words, initial pixels whose first gradients are less than 0 are selected. For an initial pixel, if the first gradient corresponding to the initial pixel is greater than 0, the first gradient corresponding to the initial pixel is modified to 0; if the first gradient corresponding to the initial pixel is less than 0, the first gradient corresponding to the initial pixel is corrected to obtain a corrected value.

[0155] In an embodiment of the present invention, if the first gradient is less than 0, a sign function in the image processor is called to correct the direction of the first gradient to obtain a second gradient.

[0156] In the embodiment of the present invention, the positive or negative value of the first gradient does not represent the direction of gradient change. If the first gradient is less than 0, the second gradient is determined according to the sign function, as shown in formula (10).

[0157]

[0158] in, is the second gradient, sign(·) is the sign function, when () is greater than 0, sign(·) takes the value of 1, when () is less than 0, sign(·) takes the value of -1, is the first target vector corresponding to the initial pixel point, is the second target vector corresponding to the initial pixel. Calling the sign function in the image processor to correct the direction of the first gradient can speed up the operation.

[0159] In an embodiment of the present invention, a normalization function is used to normalize the second gradient to obtain a third gradient, and a correction coefficient is determined according to a preset edge intensity coefficient and the preset super-resolution multiple; and the correction value is obtained according to the third gradient and the correction coefficient.

[0160] When the image super-resolution processing method is executed by an image processor, a multiplication-addition function in the image processor may be called to calculate the product of the third gradient and the correction coefficient to obtain the correction value.

[0161] Specifically, the third gradient can be obtained by formula (11).

[0162]

[0163] in, It is the third gradient, is the second gradient, normalize(·) is a normalization function, and calling the over-normalization function in the image processor to normalize the second gradient can speed up the operation.

[0164] The preset edge strength coefficient is considered a super parameter, S = strength * s, where strength is the edge strength coefficient, s is the super-resolution factor, and S is the correction coefficient. The correction value can be obtained by multiplying the correction coefficient by the third gradient, as shown in formula (12).

[0165]

[0166] in, is the correction value, S is the correction coefficient, It is the third gradient.

[0167] S32. Determine the super-resolution position corresponding to the initial pixel point according to the correction value and the super-resolution multiple.

[0168] In an embodiment of the present invention, the initial horizontal coordinate and initial vertical coordinate of the initial pixel point are obtained; a first difference between the initial horizontal coordinate and the correction value is calculated, and the horizontal coordinate of the super-resolution position is determined based on the first difference and the super-resolution multiple; a second difference between the initial vertical coordinate and the correction value is calculated, and the vertical coordinate of the super-resolution position is determined based on the second difference and the super-resolution multiple, as shown in Formula (13) and Formula (14).

[0169]

[0170]

[0171] Wherein, i′ is the horizontal coordinate of the super-resolution position, j′ is the vertical coordinate of the super-resolution position, i is the horizontal coordinate of the initial pixel point, and j is the vertical coordinate of the super-resolution position.

[0172] S34. Determine the super-resolved pixel value corresponding to the super-resolved position according to the image to be processed.

[0173] In an embodiment of the present invention, the pixel value of the super-resolved position with coordinates (i′, j′) is determined in the image to be processed. If (i′, j′) is not an integer, the super-resolved pixel value corresponding to the super-resolved position can be determined by bilinear interpolation or nearest neighbor interpolation, as shown in formula (15).

[0174]

[0175] Among them, Y i,j It is the super-resolution pixel value corresponding to the initial pixel point with coordinates (i, j).

[0176] S4. Adjust the enlarged image according to the super-resolution pixel values corresponding to the initial pixel points to obtain a super-resolution image corresponding to the image to be processed.

[0177] In the embodiment of the present invention, for each initial pixel point, the target pixel value corresponding to the initial pixel point is replaced by the super-resolution value corresponding to the initial pixel point to obtain a super-resolution image corresponding to the image to be processed.

[0178] In an embodiment of the present invention, the initial pixel point is a pixel point in the enlarged image. For each initial pixel point, the super-resolution pixel value corresponding to the initial pixel point is determined through step S3, and the target pixel value corresponding to the initial pixel point is replaced with the super-resolution pixel value corresponding to the initial pixel point to obtain a super-resolution image.

[0179] For example, for initial pixel f1, the initial pixel value of f1 is g1, and the super-resolution pixel value corresponding to f1 is G1. The pixel value of f1 is set to G1. The above operation is performed for each initial pixel in the enlarged image. In other words, the initial pixels of the enlarged image include: f1, f2, f3, ... fn. For each initial pixel in f1, f2, f3, ... fn, the pixel value of the initial pixel is set to the super-resolution pixel value corresponding to the initial pixel to obtain the super-resolution image.

[0180] In an embodiment of the present invention, an image to be processed is obtained, and an enlarged image corresponding to the image to be processed is determined based on the image to be processed and a preset super-resolution multiple; for each initial pixel in the enlarged image, a reference vector set corresponding to the initial pixel is determined, wherein the reference vector set is used to reflect the pixel values of each pixel in the neighborhood corresponding to the initial pixel in the enlarged image; for each initial pixel, a correction value corresponding to the initial pixel is determined based on the reference vector set corresponding to the initial pixel, and a super-resolution pixel value corresponding to the initial pixel is determined based on the correction value and the image to be processed; the enlarged image is adjusted based on the super-resolution pixel values corresponding to each initial pixel to obtain a super-resolution image corresponding to the image to be processed. The present invention does not use a deep learning method to generate a super-resolution image, but instead determines a reference vector set corresponding to each initial pixel, determines the correction value corresponding to the initial pixel through the reference vector set, and then determines the super-resolution pixel value corresponding to the initial pixel. In other words, the process of generating super-resolution images in the present invention does not involve complex calculations; only simple operations are required to obtain super-resolution images. The computing power required by the present invention is far less than that required by deep learning methods to generate super-resolution images, and super-resolution images can be quickly generated even in products with low computing power. For example, the Mali Bifrost GPU achieved 30ms per frame, or over 33 FPS, when super-resolutioning to 1080P on a Mali-G52. This high computing speed allows for rapid generation of super-resolution images, and thus enables real-time video super-resolution in products with low computing power.

[0181] Based on the above-mentioned image super-resolution processing method, the embodiment of the present invention further provides an image super-resolution processing device, see Figure 2 ,include:

[0182] an enlarged image determining unit, configured to obtain an image to be processed, and determine an enlarged image corresponding to the image to be processed according to the image to be processed and a preset super-resolution factor;

[0183] a reference vector set determining unit, configured to determine, for each initial pixel point in the enlarged image, a reference vector set corresponding to the initial pixel point, wherein the reference vector set is configured to reflect pixel values of each pixel point in a neighborhood corresponding to the initial pixel point in the enlarged image;

[0184] a super-resolution pixel value determining unit, configured to determine, for each initial pixel point, a correction value corresponding to the initial pixel point according to a reference vector set corresponding to the initial pixel point, and determine a super-resolution pixel value corresponding to the initial pixel point according to the correction value and the image to be processed;

[0185] The super-resolution image generating unit is used to adjust the amplified image according to the super-resolution pixel values corresponding to the initial pixel points to obtain the super-resolution image corresponding to the image to be processed.

[0186] Based on the above-mentioned image super-resolution processing method, an embodiment of the present invention further provides a computer device, which can be a terminal, and the internal structure is as follows: Figure 3 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, an image super-resolution processing method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0187] Those skilled in the art will understand that Figure 3 What is shown is only a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0188] An embodiment of the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0189] Acquire an image to be processed, and determine an enlarged image corresponding to the image to be processed according to the image to be processed and a preset super-resolution factor;

[0190] For each initial pixel point in the enlarged image, determining a reference vector set corresponding to the initial pixel point, wherein the reference vector set is used to reflect the pixel values of each pixel point in a neighborhood corresponding to the initial pixel point in the enlarged image;

[0191] For each initial pixel point, determining a correction value corresponding to the initial pixel point according to a reference vector set corresponding to the initial pixel point, and determining a super-resolution pixel value corresponding to the initial pixel point according to the correction value and the image to be processed;

[0192] The enlarged image is adjusted according to the super-resolution pixel values corresponding to the initial pixel points to obtain a super-resolution image corresponding to the image to be processed.

[0193] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0194] Acquire an image to be processed, and determine an enlarged image corresponding to the image to be processed according to the image to be processed and a preset super-resolution factor;

[0195] For each initial pixel point in the enlarged image, determining a reference vector set corresponding to the initial pixel point, wherein the reference vector set is used to reflect the pixel values of each pixel point in a neighborhood corresponding to the initial pixel point in the enlarged image;

[0196] For each initial pixel point, determining a correction value corresponding to the initial pixel point according to a reference vector set corresponding to the initial pixel point, and determining a super-resolution pixel value corresponding to the initial pixel point according to the correction value and the image to be processed;

[0197] The enlarged image is adjusted according to the super-resolution pixel values corresponding to the initial pixel points to obtain a super-resolution image corresponding to the image to be processed.

[0198] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0199] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A method for image super-resolution processing, characterized in that: include: Acquire an image to be processed, and determine, based on the image to be processed and a preset super-resolution factor, an enlarged image corresponding to the image to be processed, wherein the image to be processed is an image frame subsequent to the image frame currently being played in an image frame sequence corresponding to the playing video; For each initial pixel point in the enlarged image, determining a reference vector set corresponding to the initial pixel point, wherein the reference vector set is used to reflect the pixel values of each pixel point in a neighborhood corresponding to the initial pixel point in the enlarged image; For each initial pixel point, determining a correction value corresponding to the initial pixel point according to a reference vector set corresponding to the initial pixel point, and determining a super-resolution pixel value corresponding to the initial pixel point according to the correction value and the image to be processed; Adjusting the amplified image according to the super-resolution pixel values corresponding to the initial pixel points to obtain a super-resolution image corresponding to the image to be processed; Determining, for each initial pixel point in the enlarged image, a reference vector set corresponding to the initial pixel point includes: determining an initial vector set by a processor that executes the image super-resolution method based on an initial pixel value corresponding to the initial pixel point and each reference pixel value corresponding to each reference pixel point, and determining a reference vector set corresponding to the initial pixel point based on the initial vector set, wherein the displacement between each reference pixel point and the initial pixel point is equal.

2. The image super-resolution processing method according to claim 1, characterized in that: Determining the reference vector set corresponding to the initial pixel point specifically includes: Determine several reference pixel points corresponding to the initial pixel point, and obtain a reference pixel value corresponding to each reference pixel point and an initial pixel value corresponding to the initial pixel point; An initial vector set is determined according to the obtained initial pixel value and each reference pixel value, and a reference vector set corresponding to the initial pixel point is determined according to the initial vector set.

3. The image super-resolution processing method according to claim 2, characterized in that: The plurality of reference pixel points include a first reference pixel point, a second reference pixel point, a third reference pixel point, and a fourth reference pixel point; and determining the plurality of reference pixel points corresponding to the initial pixel point specifically includes: Acquire a first reference pixel point and a second reference pixel point located in a row direction of the initial pixel point; Acquire a third reference pixel point and a fourth reference pixel point located in a direction of the initial pixel point column.

4. The image super-resolution processing method according to claim 3, characterized in that: The first reference pixel is located to the left of the initial pixel, the second reference pixel is located to the right of the initial pixel, the third reference pixel is located below the initial pixel, and the fourth reference pixel is located above the initial pixel; the image to be processed is in YUV format, and the first reference pixel value is the Y component value corresponding to the first reference pixel; The second reference pixel value is the Y component value corresponding to the second reference pixel point; The third reference pixel value is the Y component value corresponding to the third reference pixel point, and the fourth reference pixel value is the Y component value corresponding to the fourth reference pixel point.

5. The image super-resolution processing method according to claim 4, characterized in that: The determining of an initial vector set according to the obtained initial pixel value and each reference pixel value, and determining a reference vector set corresponding to the initial pixel point according to the initial vector set, specifically includes: Converting the initial pixel value into a central initial vector, converting the first reference pixel value corresponding to the first reference pixel into a first initial vector, converting the second reference pixel value corresponding to the second reference pixel into a second initial vector, converting the third reference pixel value corresponding to the third reference pixel into a third initial vector, and converting the fourth reference pixel value corresponding to the fourth reference pixel into a fourth initial vector, to obtain an initial vector set; A first target vector is determined according to the first initial vector and the third initial vector, a second target vector is determined according to the second initial vector and the fourth initial vector, and a third target vector is determined according to the center initial vector to obtain a reference vector set.

6. The image super-resolution processing method according to claim 5, characterized in that: The determining the correction value corresponding to the initial pixel point according to the reference vector set corresponding to the initial pixel point specifically includes: Calculating a first gradient according to the reference vector set; If the first gradient is less than 0, the first gradient is corrected according to the super-resolution factor to obtain a corrected value.

7. The image super-resolution processing method according to claim 6, characterized in that: The calculating the first gradient according to the reference vector set specifically includes: Calculating a sum of the first target vector and the second target vector to obtain an intermediate vector; A first gradient is calculated using a multiplication-addition function according to the intermediate vector and the third target vector.

8. The image super-resolution processing method according to claim 6, characterized in that: If the first gradient is less than 0, the first gradient is corrected according to the super-resolution factor to obtain a correction value, specifically including: If the first gradient is less than 0, calling a sign function in the processor to correct the direction of the first gradient to obtain a second gradient; Normalizing the second gradient using a normalization function to obtain a third gradient; Determining a correction coefficient according to a preset edge strength coefficient and the preset super-resolution multiple; The correction value is obtained according to the third gradient and the correction coefficient.

9. The image super-resolution processing method according to any one of claims 5 to 8, characterized in that: The image super-resolution processing method is applied to an image processor, the dimensions of the first target vector, the second target vector, and the third target vector are all configured to be equal to the word length of the operation unit of the image processor, and the dimensions of the first initial vector, the second initial vector, the third initial vector, the fourth initial vector, and the center initial vector are all configured to be half the length of the word length.

10. The image super-resolution processing method according to claim 1, characterized in that: The step of determining the super-resolution pixel value corresponding to the initial pixel point according to the correction value and the image to be processed specifically includes: Determine the super-resolution position corresponding to the initial pixel point according to the correction value and the super-resolution multiple; The super-resolution pixel value corresponding to the super-resolution position is determined according to the image to be processed.

11. The image super-resolution processing method according to claim 10, characterized in that: The determining the super-resolution position corresponding to the initial pixel point according to the correction value and the super-resolution multiple specifically includes: Obtaining the initial horizontal coordinate and initial vertical coordinate of the initial pixel point; Calculating a first difference between the initial horizontal coordinate and the correction value, and determining the horizontal coordinate of the super-resolution position according to the first difference and the super-resolution multiple; A second difference between the initial ordinate and the correction value is calculated, and the ordinate of the super-resolution position is determined according to the second difference and the super-resolution multiple.

12. The image super-resolution processing method according to claim 1, characterized in that: The step of determining, based on the image to be processed and a preset super-resolution factor, an enlarged image corresponding to the image to be processed specifically includes: Determine a number of initial pixel points according to the image to be processed and a preset super-resolution multiple; For each initial pixel point, determining an initial pixel value of the initial pixel point according to the image to be processed; An enlarged image is determined according to a plurality of initial pixel points and initial pixel values respectively corresponding to the plurality of initial pixel points.

13. The image super-resolution processing method according to claim 12, characterized in that: The step of determining, for each initial pixel point, an initial pixel value of the initial pixel point according to the image to be processed specifically includes: For each initial pixel point, determining a first position and several second positions corresponding to the initial pixel point in the image to be processed, wherein the coordinates of the initial pixel point are the product of the coordinates of the first position and the super-resolution multiple, and the displacement between the coordinates of each second position and the coordinates of the first position is equal; Obtaining a first pixel value corresponding to the first position and second pixel values corresponding to each of the plurality of second positions; An average pixel value of the first pixel value and each second pixel value is determined, and the average pixel value is used as the initial pixel value.

14. The image super-resolution processing method according to claim 13, characterized in that: The determining of the first position and the plurality of second positions corresponding to the initial pixel point in the image to be processed specifically includes: Determining a first position corresponding to the initial pixel point in the image to be processed; The plurality of second positions are determined in the image to be processed according to the first position and a preset expansion value, wherein a displacement between each second position and the first position is equal to the expansion value.

15. The image super-resolution processing method according to claim 1, characterized in that: The step of adjusting the enlarged image according to the super-resolution pixel values corresponding to the initial pixel points to obtain the super-resolution image corresponding to the image to be processed specifically includes: For each initial pixel point, the target pixel value corresponding to the initial pixel point is replaced by the super-resolution value corresponding to the initial pixel point to obtain a super-resolution image corresponding to the image to be processed.

16. An image super-resolution processing device, characterized in that: include: an enlarged image determining unit, configured to obtain an image to be processed and determine an enlarged image corresponding to the image to be processed based on the image to be processed and a preset super-resolution factor, wherein the image to be processed is an image frame subsequent to the image frame currently being played in the image frame sequence corresponding to the playing video; a reference vector set determining unit, configured to determine, for each initial pixel point in the enlarged image, a reference vector set corresponding to the initial pixel point, wherein the reference vector set is configured to reflect pixel values of each pixel point in a neighborhood corresponding to the initial pixel point in the enlarged image; a super-resolution pixel value determining unit, configured to determine, for each initial pixel point, a correction value corresponding to the initial pixel point according to a reference vector set corresponding to the initial pixel point, and determine a super-resolution pixel value corresponding to the initial pixel point according to the correction value and the image to be processed; a super-resolution image generating unit, configured to adjust the amplified image according to the super-resolution pixel values corresponding to the initial pixel points, so as to obtain a super-resolution image corresponding to the image to be processed; Determining, for each initial pixel point in the enlarged image, a reference vector set corresponding to the initial pixel point includes: determining an initial vector set by a processor that executes the image super-resolution method based on an initial pixel value corresponding to the initial pixel point and each reference pixel value corresponding to each reference pixel point, and determining a reference vector set corresponding to the initial pixel point based on the initial vector set, wherein the displacement between each reference pixel point and the initial pixel point is equal.

17. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 15 are implemented.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.

Images