Image super-resolution edge processing method, apparatus and readable storage medium

Abstract

This application relates to an image super-resolution edge processing method, apparatus, and readable storage medium, belonging to the field of image processing technology. The method includes: calculating the derivative of a low-resolution grayscale image to obtain a derivative matrix; segmenting the edges of the low-resolution grayscale image based on the non-zero response characteristics of the derivative values ​​of pixels in the derivative matrix to obtain multiple edge pixel segments and their corresponding edge directions; determining whether a target pixel in the low-resolution grayscale image is located within any edge pixel segment, wherein the target pixel corresponds to a pixel to be determined in the super-resolution image; if so, calculating the super-resolution pixel value of the pixel to be determined according to the target interpolation calculation formula corresponding to the edge direction of the edge pixel segment where the target pixel is located. This application improves the image super-resolution effect by performing characteristic processing on different edge types and strengthening the directional processing of edges, making the edge parts clearer.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image super-resolution edge processing method, apparatus and readable storage medium. Background Technology

[0002] With the development of digital media technology, the demand for high-definition digital images is gradually increasing. However, the storage and transmission of high-definition images require enormous resources. To address this issue, image super-resolution upscaling technology has received widespread attention. Image super-resolution upscaling is a method of obtaining a high-resolution image from a low-resolution image. It allows for the use of only the low-resolution image during storage and transmission, while the high-resolution image is obtained during display using super-resolution technology, thus greatly reducing system resource consumption. Currently, the main method for image upscaling and super-resolution is the traditional interpolation method, which is the most basic, simple, and direct method in super-resolution technology. It mainly simulates the mapping relationship between pixels through different calculation methods and uses information from surrounding pixels to interpolate the image to obtain the reconstructed result.

[0003] In related technologies, traditional interpolation methods use the weighted sum of points surrounding the pixel in the image to calculate the pixel value. However, since the weighted sum is similar to an averaging algorithm and treats each pixel equally, while the image edges are generally directional, it can cause a certain blurring effect in the super-resolution image. Currently, image enhancement and sharpening are often performed on the enlarged interpolated image to solve the image blurring caused by the interpolation method. However, since the image information of the enlarged interpolated image has been lost, image enhancement and sharpening cannot effectively solve the image blurring problem. Summary of the Invention

[0004] This application provides an image super-resolution edge processing method, apparatus, and readable storage medium to solve the edge blurring problem caused by traditional interpolation processing of the whole image in related technologies.

[0005] Firstly, an image super-resolution edge processing method is provided, comprising the following steps:

[0006] The derivative matrix is ​​obtained by performing derivative calculations on the low-resolution grayscale image.

[0007] Based on the non-zero response characteristic of the derivative values ​​of pixels in the derivative matrix, the edges of low-resolution grayscale images are segmented to obtain multiple edge pixel segments and their corresponding edge directions.

[0008] Determine whether a target pixel in a low-resolution grayscale image is located within any edge pixel segment, wherein the target pixel has a corresponding relationship with the pixel to be determined in the super-resolution image;

[0009] If so, the super-resolution pixel value of the pixel to be determined is calculated according to the target interpolation calculation formula corresponding to the edge direction of the edge pixel segment where the target pixel is located.

[0010] In some embodiments, the step of calculating the derivative of the low-resolution grayscale image to obtain the derivative matrix includes:

[0011] The first-order derivative matrix and the second-order derivative matrix are obtained by calculating the derivatives of the low-resolution grayscale image using the first-order and second-order derivative operators, respectively.

[0012] In some embodiments, the edge pixel segment includes a hard edge pixel segment and a soft edge pixel segment. The non-zero derivative response characteristic of the hard edge pixel segment is that both the first and second orders have the same brief non-zero response, and the second order has both positive and negative values. The non-zero derivative response characteristic of the soft edge pixel segment is that the first order has a continuous non-zero response, and the second order non-zero response only appears at the beginning and end of the soft edge pixel segment.

[0013] In some embodiments, the edge pixel segment where the target pixel is located is a target hard edge pixel segment, and the step of calculating the super-resolution pixel value of the pixel to be determined according to the target interpolation calculation formula corresponding to the edge direction of the edge pixel segment where the target pixel is located includes:

[0014] Construct the equation of the edge direction line based on the coordinates of the target pixel and the edge direction corresponding to the target hard edge pixel segment;

[0015] The super-resolution pixel value of the pixel to be determined is calculated based on the equation of the straight line along the edge direction and the coordinates of the nearest integer target point to the target pixel.

[0016] In some embodiments, the edge pixel segment where the target pixel is located is a target soft edge pixel segment, and the step of calculating the super-resolution pixel value of the pixel to be determined according to the target interpolation calculation formula corresponding to the edge direction of the edge pixel segment where the target pixel is located includes:

[0017] Construct the equation of the normal line based on the coordinates of the target pixel and the edge direction corresponding to the target soft edge pixel segment;

[0018] The super-resolution pixel value of the pixel to be determined is calculated based on the equation of the normal line and the coordinates of the two nearest integer points to the target pixel.

[0019] In some embodiments, the edge direction is determined based on the coordinates of the first and last pixels in the edge pixel segment, and the formula for calculating the edge direction value is as follows:

[0020] tanθ = (b2 - b1) / (a2 - a1)

[0021] In the formula, tanθ represents the edge direction value, (a1,b1) represents the coordinates of the first pixel, and (a2,b2) represents the coordinates of the last pixel.

[0022] In some embodiments, after the step of calculating the super-resolution pixel value of the pixel to be determined according to the target interpolation calculation formula corresponding to the edge direction of the edge pixel segment where the target pixel is located, the method further includes:

[0023] Edge detection is performed on the original image corresponding to the low-resolution grayscale image using an edge detection operator to obtain a binarized image.

[0024] The edge portion of the binarized image is processed based on the coordinates of each pixel in each edge pixel segment to obtain a new binarized image including multiple first pixels, where the first pixels belong to the detail portion of the image.

[0025] Based on the first pixel and the preset pixel block side length, the detail part in the new binarized image is divided into blocks to obtain multiple detail pixel blocks;

[0026] The super-resolution reconstruction of each detail pixel block is performed by using an iterative back projection method to obtain the super-resolution pixel value of each pixel in the detail pixel block.

[0027] In some embodiments, after the step of performing super-resolution reconstruction processing on each detail pixel block using an iterative back-projection method to obtain the super-resolution pixel value of each pixel in the detail pixel block, the method further includes:

[0028] The original image is super-resolution magnified using an interpolation method to obtain a magnified interpolated image;

[0029] The pixel values ​​in the magnified interpolated image are replaced based on the super-resolution pixel values ​​of the pixel to be determined and the super-resolution pixel values ​​of each pixel in the detail pixel block to generate the target super-resolution image.

[0030] Secondly, an image super-resolution edge processing apparatus is provided, comprising:

[0031] The first processing unit is used to perform derivative calculations on the low-resolution grayscale image to obtain the derivative matrix;

[0032] The second processing unit is used to segment the edges of the low-resolution grayscale image based on the non-zero response characteristics of the derivative values ​​of the pixels in the derivative matrix, and to obtain multiple edge pixel segments and their corresponding edge directions.

[0033] The third processing unit is used to determine whether the target pixel in the low-resolution grayscale image is located within any edge pixel segment, and the target pixel has a corresponding relationship with the pixel to be determined in the super-resolution image.

[0034] The fourth processing unit is used to calculate the super-resolution pixel value of the pixel to be determined according to the target interpolation calculation formula corresponding to the edge direction of the edge pixel segment where the target pixel is located, if the condition is met.

[0035] Thirdly, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, which, when executed by a computer, cause the computer to perform the aforementioned image super-resolution edge processing method.

[0036] This application provides an image super-resolution edge processing method, apparatus, and readable storage medium, including: calculating the derivative of a low-resolution grayscale image to obtain a derivative matrix; segmenting the edges of the low-resolution grayscale image based on the non-zero response characteristics of the derivative values ​​of pixels in the derivative matrix to obtain multiple edge pixel segments and their corresponding edge directions; determining whether a target pixel in the low-resolution grayscale image is located within any edge pixel segment, wherein the target pixel corresponds to a pixel to be determined in the super-resolution image; if so, calculating the super-resolution pixel value of the pixel to be determined according to the target interpolation formula corresponding to the edge direction of the edge pixel segment where the target pixel is located. This application classifies the edge parts of the image through non-zero response characteristics to perform characteristic processing on different edge types, and uses different interpolation formulas to enhance the directional processing of the edges, making the edge parts clearer, thereby solving the edge blurring problem and improving the image super-resolution effect. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 A flowchart illustrating an image super-resolution edge processing method provided in this application embodiment;

[0039] Figure 2 This is a schematic diagram illustrating the specific process of image super-resolution edge processing provided in the embodiments of this application;

[0040] Figure 3 A schematic diagram illustrating the various edge types and their corresponding first and second derivative characteristics provided in the embodiments of this application;

[0041] Figure 4 This is a schematic diagram of a step-shaped edge in the 30° direction provided in an embodiment of this application;

[0042] Figure 5 This is a schematic diagram of a sloping edge in the 45° direction provided in an embodiment of this application;

[0043] Figure 6 A schematic diagram showing the segmentation of details provided in the embodiments of this application. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0045] This application provides an image super-resolution edge processing method, apparatus, and readable storage medium, which can solve the edge blurring problem caused by traditional interpolation processing of the whole image in related technologies.

[0046] See Figure 1 and Figure 2 As shown, this application provides an image super-resolution edge processing method, including the following steps:

[0047] Step S10: Calculate the derivative of the low-resolution grayscale image to obtain the derivative matrix;

[0048] As an example, in this embodiment, a low-resolution image (e.g., an image with a resolution of 256x256) will be denoised and converted to grayscale using filtering methods such as Gaussian filtering to obtain a low-resolution grayscale image. Here, (x,y) can represent the coordinates of a pixel in the low-resolution grayscale image, and f(x,y) can represent the grayscale value of the corresponding pixel. Therefore, after obtaining the low-resolution grayscale image, point-by-point operations will be performed on the grayscale value f(x,y) corresponding to the pixel (x,y) in the low-resolution grayscale image to obtain the derivative matrix.

[0049] Furthermore, the step of calculating the derivative of the low-resolution grayscale image to obtain the derivative matrix includes:

[0050] The first-order derivative matrix and the second-order derivative matrix are obtained by calculating the derivatives of the low-resolution grayscale image using the first-order and second-order derivative operators, respectively.

[0051] As an example, in this embodiment, a low-resolution grayscale image can be processed using first-order and second-order derivative operators respectively to obtain first-order and second-order derivative matrices of the same size. For example, for an image with a resolution of 256×256, both matrices will be the same size, 256×256. It should be understood that this embodiment calculates the first and second derivatives for each pixel in both the horizontal and vertical directions. Therefore, each pixel has corresponding first-order and second-order derivative values ​​in both the X and Y axes. That is, for a low-resolution grayscale image, it has two first-order derivative matrices and two second-order derivative matrices.

[0052] It is understandable that pixels with non-zero derivative values ​​possess a "non-zero response." In this embodiment, for ease of detection, a "non-zero response" is defined as the portion of the derivative threshold that is higher than the grayscale level M / 64, rounded down. A "continuous non-zero response" is defined as the portion of pixels with consecutive non-zero derivative values ​​for N / 100 pixels of the original image's horizontal resolution, where M refers to the grayscale level and N refers to the horizontal resolution. For example, for an image with a resolution of 1280×720 and 256 grayscale levels, its "non-zero response" is 256 / 64 = 4. Therefore, pixels in the matrix with derivative values ​​higher than 4 possess a "non-zero response." A "continuous non-zero response" is 1280 / 100 = 12.8, rounded down to 12. Therefore, if there are 12 consecutive pixels in the matrix with "non-zero responses," then that portion possesses a "continuous non-zero response." Thus, edge features can be identified by comparing the derivative features corresponding to each pixel in the first-order and second-order derivative matrices.

[0053] Furthermore, the edge pixel segment includes a hard edge pixel segment and a soft edge pixel segment. The non-zero derivative response characteristic of the hard edge pixel segment is that both the first and second orders have the same brief non-zero response, and the second order has both positive and negative values. The non-zero derivative response characteristic of the soft edge pixel segment is that the first order has a continuous non-zero response, and the second order non-zero response only appears at the beginning and end of the soft edge pixel segment.

[0054] As an example, in this embodiment, edge types will be defined, specifically including hard edges and soft edges. Hard edges include step edges and pulse edges, while soft edges include ramp edges and ridge edges. It should be understood that hard edges and soft edges can be distinguished by the non-zero response characteristics of their derivative values. Specifically, (1) step edges and pulse edges (hard edges): the first and second orders have the same response, and the second order has a process from positive to negative; (2) ramp edges and ridge edges (soft edges): the first order has a continuous non-zero response, and the second order non-zero response only appears at the start and end points.

[0055] Step S20: Based on the non-zero response characteristics of the derivative values ​​of pixels in the derivative matrix, the edges of the low-resolution grayscale image are segmented to obtain multiple edge pixel segments and their corresponding edge directions;

[0056] As an example, in this implementation, see Figure 3 As shown, the edge direction will be determined, and specific pixels will be processed to classify the edges and identify edge pixel segments. The following explanation will use the processing from the top-left pixel towards the bottom-right pixel as an example.

[0057] The algorithm determines whether a pixel in a low-resolution grayscale image exhibits a non-zero response in either of its first derivatives along two directions. If neither first derivative shows a non-zero response, the pixel is considered an edge that does not require processing. Otherwise, it continues to investigate whether there are consecutive non-zero responses between the pixel and other pixels based on the first derivatives of other pixels. It should be noted that if a pixel exhibits a non-zero response in only one direction, the algorithm continues to explore for consecutive non-zero responses along the direction with the strongest response. Conversely, if both first derivatives show non-zero responses, the algorithm continues to detect consecutive non-zero responses along the direction with the strongest response.

[0058] If a pixel is detected to have a continuous non-zero response with other pixels, i.e., a pixel block with a continuous non-zero response is detected, the second derivative of each pixel in the pixel block will continue to be detected. When only the first and last pixels in the pixel block have a non-zero response in their second derivatives, the pixel block will be determined to be a soft edge in that direction, and the pixel block will constitute a soft edge pixel segment in that direction; otherwise, the pixel block will be determined to be an edge part that does not need to be processed.

[0059] If a pixel is detected to have no continuous non-zero response with other pixels, that is, if the non-zero response of the first derivative of each pixel in the pixel block that does not have a continuous non-zero response is very brief, then it is necessary to continue to determine whether the second derivative of the pixels in the pixel block has the same brief non-zero response and has both positive and negative values. If so, the pixel block is determined to be a hard edge part in that direction, and the pixel block will constitute a hard edge pixel segment in that direction; otherwise, it is an edge part that does not need to be processed.

[0060] This completes the detection of a single pixel. The process then moves on to determine if the next pixel is already detected. If it is, the detection is skipped; otherwise, the detection continues using the same method. As can be seen, this process yields multiple edge pixel segments in different directions. The edge direction value can be determined based on the first and last pixels of each edge segment.

[0061] Furthermore, the edge direction is determined based on the coordinates of the first and last pixels in the edge pixel segment, and the formula for calculating the edge direction value is as follows:

[0062] tanθ = (b2 - b1) / (a2 - a1)

[0063] In the formula, tanθ represents the edge direction value, (a1,b1) represents the coordinates of the first pixel, and (a2,b2) represents the coordinates of the last pixel.

[0064] As an example, in this embodiment, for a certain edge pixel segment, the edge direction value tanθ of the edge pixel segment can be calculated by substituting the coordinates of its first and last pixels into the following formula:

[0065] tanθ = (b2 - b1) / (a2 - a1)

[0066] The edge direction corresponding to the edge pixel segment can be determined by the edge direction value tanθ mentioned above. Figure 4 The step edge in the 30° direction shown and Figure 5 The sloping edge in the 45° direction shown.

[0067] Step S30: Determine whether the target pixel in the low-resolution grayscale image is located within any edge pixel segment, wherein the target pixel has a corresponding relationship with the pixel to be determined in the super-resolution image;

[0068] As an example, in this embodiment, the determination of whether the pixel to be processed belongs to the edge region needs to be processed is made by judging whether the target pixel corresponding to the pixel to be processed in the low-resolution grayscale image is located within any edge pixel segment. If it belongs to the edge region to be processed, its edge type and edge direction can be determined by the edge pixel segment it is located in. For example, assuming the image super-resolution factor is r and the coordinates of the pixel to be processed after super-resolution are (X, Y), then the coordinates of the target pixel corresponding to the pixel to be processed in the original low-resolution grayscale image are (X / r, Y / r). At this time, it is necessary to determine whether the target pixel (X / r, Y / r) is within the edge pixel segment.

[0069] Step S40: If yes, then calculate the super-resolution pixel value of the pixel to be determined according to the target interpolation calculation formula corresponding to the edge direction of the edge pixel segment where the target pixel is located.

[0070] As an example, in this embodiment, if it is confirmed that the target pixel corresponding to the pixel to be determined is not within any edge pixel segment, it indicates that the pixel to be determined belongs to an edge part that does not need to be processed, and therefore no further processing is performed. However, if it is confirmed that the target pixel corresponding to the pixel to be determined is within a certain edge pixel segment, it indicates that the pixel to be determined belongs to an edge part that needs to be processed, and then the super-resolution pixel value of the pixel to be determined needs to be calculated according to the interpolation calculation formula corresponding to the edge direction of the edge pixel segment where the target pixel is located. It should be noted that this embodiment sets different interpolation calculation formulas for different edge directions, so as to calculate the super-resolution pixel value of pixels in different edge directions through different interpolation calculation formulas.

[0071] Therefore, this embodiment classifies the edge parts of the image by using non-zero response characteristics to perform characteristic processing on different edge types. At the same time, it uses different interpolation calculation formulas to enhance the directional processing of the edges, making the edge parts clearer, thereby solving the edge blurring problem and improving the image super-resolution effect.

[0072] Furthermore, the edge pixel segment where the target pixel is located is a target hard edge pixel segment, and the step of calculating the super-resolution pixel value of the pixel to be determined according to the target interpolation calculation formula corresponding to the edge direction of the edge pixel segment where the target pixel is located includes:

[0073] Construct the equation of the edge direction line based on the coordinates of the target pixel and the edge direction corresponding to the target hard edge pixel segment;

[0074] The super-resolution pixel value of the pixel to be determined is calculated based on the equation of the straight line along the edge direction and the coordinates of the nearest integer target point to the target pixel.

[0075] As an example, in this embodiment, for hard edges such as step and pulse types, the relevant portion of the edge will be directly magnified by a factor of magnification. Specifically, assuming the image super-resolution factor is r, the coordinates of the pixel to be determined after super-resolution are (X, Y), and the coordinates of the target pixel corresponding to the pixel to be determined in the original low-resolution grayscale image are (X / r, Y / r), then based on the edge direction value corresponding to the edge pixel segment where the target pixel is located and passing through the point (X / r, Y / r), establish the equation of the edge direction line: y = (xX / r)·tanθ + Y / r; then detect the nearest integer point on this line (i.e., take the nearest point along the edge direction on the super-resolution image), and use the pixel value of this point as the super-resolution pixel value of the pixel to be determined. Specifically, the target integer point (x0, y0) that satisfies the above edge direction line equation can be taken, and the target integer point (x0, y0) is the integer point that minimizes the distance l = (x0 - X / r)^2 + (y0 - Y / r)^2; then, use the pixel value of the detected target integer point as the super-resolution pixel value of the pixel to be determined, that is, fill the pixel value of the pixel to be determined with the pixel value of the target integer point.

[0076] For example, suppose the coordinates of the pixel to be determined are (33, 60) and the super-resolution factor is 2. Then the coordinates of the target pixel in the original low-resolution grayscale image are (16.5, 30). Suppose the edge direction value tanθ = 2 corresponding to the hard edge pixel segment where the target pixel is located. Then the equation of the edge direction line passing through this target pixel is y = 2(x-16.5) + 30. At this time, through calculation, it can be seen that the integer point that satisfies the edge direction line equation and minimizes the distance l = (x0-16.5)^2 + (y0-30)^2 is (16, 29). Therefore, the integer point (16, 29) is taken as the target integer point, and the pixel value of the target integer point (16, 29) is used to fill the pixel value of the pixel to be determined.

[0077] Furthermore, the edge pixel segment where the target pixel is located is a target soft edge pixel segment, and the step of calculating the super-resolution pixel value of the pixel to be determined according to the target interpolation calculation formula corresponding to the edge direction of the edge pixel segment where the target pixel is located includes:

[0078] Construct the equation of the normal line based on the coordinates of the target pixel and the edge direction corresponding to the target soft edge pixel segment;

[0079] The super-resolution pixel value of the pixel to be determined is calculated based on the equation of the normal line and the coordinates of the two nearest integer points to the target pixel.

[0080] As an example, in this embodiment, for soft edges such as sloping and ridge-shaped edges, equation creation and integer point determination are also required. Specifically, assuming the image super-resolution factor is r, the coordinates of the super-resolution pixel to be determined are (X,Y), and the coordinates of the target pixel corresponding to the pixel to be determined in the original low-resolution grayscale image are (X / r,Y / r), then a normal line equation is established based on the edge direction value corresponding to the edge pixel segment where the target pixel is located and passing through the point (X / r,Y / r): y=(xX / r)·(tanθ+π / 2)+Y / r. At the same time, the two target integer points (x1,y1) and (x2,y2) closest to the target pixel (X / r,Y / r) on this line equation are taken. Then, the super-resolution pixel value of the pixel to be determined can be calculated based on the pixel values ​​g(x1,y1) and g(x2,y2) of the two target integer points. The specific calculation formula for the super-resolution pixel value g(X / r,Y / r) is as follows:

[0081]

[0082] It should be understood that under grayscale (i.e., single-channel) conditions, the grayscale value is equal to the pixel value, while in color images (i.e., three channels: R, G, and B), the grayscale value is not equal to the pixel value.

[0083] Furthermore, after the step of calculating the super-resolution pixel value of the pixel to be determined according to the target interpolation calculation formula corresponding to the edge direction of the edge pixel segment where the target pixel is located, the method further includes:

[0084] Edge detection is performed on the original image corresponding to the low-resolution grayscale image using an edge detection operator to obtain a binarized image.

[0085] The edge portion of the binarized image is processed based on the coordinates of each pixel in each edge pixel segment to obtain a new binarized image including multiple first pixels, where the first pixels belong to the detail portion of the image.

[0086] Based on the first pixel and the preset pixel block side length, the detail part in the new binarized image is divided into blocks to obtain multiple detail pixel blocks;

[0087] The super-resolution reconstruction of each detail pixel block is performed by using an iterative back projection method to obtain the super-resolution pixel value of each pixel in the detail pixel block.

[0088] As an example, in this embodiment, the details of the image will also be segmented to form easily processed pixel blocks. Reconstruction or deep learning methods will then be used to process these pixel blocks, resulting in better reconstruction of the details. Specifically, edge detection operators such as Canny can be preferably used to perform edge detection on the original image. The detection result is a binary image where non-edge and non-detail portions are assigned a value of 0, and edge and detail portions are assigned a value of 1. Then, according to the coordinates of each pixel in the hard and soft edge pixel segments determined in the aforementioned steps, the corresponding soft and hard edge portions in the binary image are removed, i.e., their values ​​are changed from 1 to 0, to form a new binary image. It is understood that, see [link to previous section]. Figure 6 As shown, the new binarized image retains only the more cluttered details, that is, the pixels assigned a value of 1 in the new binarized image (i.e., the first pixel) belong to the details.

[0089] Next, starting from the first pixel in the upper left corner of the new binarized image, detection is performed point by point. The first pixel with a value of 1 is the detail part. Then, pixel block segmentation is performed with the first pixel as the center and a preset pixel block side length. For example, if the vertical resolution of the original image is N, and the pixel block side length is n = N / 100 (for example, n can be 3, 4, or 5, etc.), then a pixel block consisting of n×n pixels around the first pixel is taken as a processing unit. Then, detection continues. If the next pixel with a value of 1 is in the already detected pixel block, it is skipped; if not, an n×n pixel block is taken with that point as the center. This process continues until the detail part is segmented into multiple n×n small pixel blocks. Finally, these pixel blocks are processed using reconstruction or learning methods. For example, iterative back projection is preferred to perform super-resolution reconstruction processing on each detail pixel block to obtain the super-resolution pixel value of each pixel in the detail pixel block.

[0090] Therefore, this embodiment can effectively improve computational efficiency by dividing the detailed parts into blocks and then using an iterative back projection method on the blocks.

[0091] Furthermore, after the step of performing super-resolution reconstruction processing on each detail pixel block using an iterative back-projection method to obtain the super-resolution pixel value of each pixel in the detail pixel block, the method further includes:

[0092] The original image is super-resolution magnified using an interpolation method to obtain a magnified interpolated image;

[0093] The pixel values ​​in the magnified interpolated image are replaced based on the super-resolution pixel values ​​of the pixel to be determined and the super-resolution pixel values ​​of each pixel in the detail pixel block to generate the target super-resolution image.

[0094] In this exemplary embodiment, after super-resolution reconstruction of the edge and detail portions, the final super-resolution image is reconstructed based on the reconstructed edge and detail portions. Specifically, bicubic interpolation or other interpolation methods can be preferred to enlarge the original image to obtain an enlarged interpolated image, such as a 512×512 enlarged interpolated image to be processed. Then, based on the coordinates of each pixel in the soft and hard edge pixel segments determined in the aforementioned steps, the coordinates of the pixel points in the enlarged interpolated image that need to have their pixel values ​​replaced are determined. Finally, the super-resolution pixel values ​​of each pixel in the soft and hard edge pixel segments are replaced with the corresponding pixel points in the enlarged interpolated image according to the pixel coordinate values, thereby generating the final super-resolution image. Thus, this embodiment fuses the edge super-resolution results and detail super-resolution results obtained through two processing methods to obtain a super-resolution image with better super-resolution performance.

[0095] In summary, this embodiment classifies the edge regions of an image based on non-zero response characteristics, applying different processing methods to edges with different attributes. Furthermore, the interpolation calculation formula is adjusted according to the edge direction, making the edges clearer. In addition, experiments were conducted using traditional nearest neighbor interpolation, bicubic interpolation, and the method described in this embodiment. The resulting PSNR (Peak Signal-to-Noise Ratio) values ​​after super-resolution were 27.6, 28.4, and 30.1, respectively. This demonstrates that the optimization of super-resolution performance in this embodiment is effective and feasible.

[0096] This embodiment further segments the detailed parts and uses an iterative back-projection method for super-resolution reconstruction of these segments. Compared with the traditional deep learning ESNR method, this effectively optimizes computational efficiency. For example, in an environment with an Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz and 4.00GB of memory, the time taken to perform a 2x super-resolution on a 256×256 image using traditional deep learning methods and this embodiment is 5.456s and 0.892s respectively. This demonstrates that this embodiment effectively improves computational efficiency.

[0097] This application also provides an image super-resolution edge processing apparatus, including:

[0098] The first processing unit is used to perform derivative calculations on the low-resolution grayscale image to obtain the derivative matrix;

[0099] The second processing unit is used to segment the edges of the low-resolution grayscale image based on the non-zero response characteristics of the derivative values ​​of the pixels in the derivative matrix, and to obtain multiple edge pixel segments and their corresponding edge directions.

[0100] The third processing unit is used to determine whether the target pixel in the low-resolution grayscale image is located within any edge pixel segment, and the target pixel has a corresponding relationship with the pixel to be determined in the super-resolution image.

[0101] The fourth processing unit is used to calculate the super-resolution pixel value of the pixel to be determined according to the target interpolation calculation formula corresponding to the edge direction of the edge pixel segment where the target pixel is located, if the condition is met.

[0102] Furthermore, the first processing unit is specifically used for:

[0103] The first-order derivative matrix and the second-order derivative matrix are obtained by calculating the derivatives of the low-resolution grayscale image using the first-order and second-order derivative operators, respectively.

[0104] Furthermore, the edge pixel segment includes a hard edge pixel segment and a soft edge pixel segment. The non-zero derivative response characteristic of the hard edge pixel segment is that both the first and second orders have the same brief non-zero response, and the second order has both positive and negative values. The non-zero derivative response characteristic of the soft edge pixel segment is that the first order has a continuous non-zero response, and the second order non-zero response only appears at the beginning and end of the soft edge pixel segment.

[0105] Furthermore, the fourth processing unit is specifically used for:

[0106] Construct the equation of the edge direction line based on the coordinates of the target pixel and the edge direction corresponding to the target hard edge pixel segment;

[0107] The super-resolution pixel value of the pixel to be determined is calculated based on the equation of the straight line along the edge direction and the coordinates of the nearest integer target point to the target pixel.

[0108] Furthermore, the edge pixel segment where the target pixel point is located is the target soft edge pixel segment, and the fourth processing unit is specifically used for:

[0109] Construct the equation of the normal line based on the coordinates of the target pixel and the edge direction corresponding to the target soft edge pixel segment;

[0110] The super-resolution pixel value of the pixel to be determined is calculated based on the equation of the normal line and the coordinates of the two nearest integer points to the target pixel.

[0111] Furthermore, the edge direction is determined based on the coordinates of the first and last pixels in the edge pixel segment, and the formula for calculating the edge direction value is as follows:

[0112] tanθ = (b2 - b1) / (a2 - a1)

[0113] In the formula, tanθ represents the edge direction value, (a1,b1) represents the coordinates of the first pixel, and (a2,b2) represents the coordinates of the last pixel.

[0114] Furthermore, the device also includes a fifth processing unit, which is used for:

[0115] Edge detection is performed on the original image corresponding to the low-resolution grayscale image using an edge detection operator to obtain a binarized image.

[0116] The edge portion of the binarized image is processed based on the coordinates of each pixel in each edge pixel segment to obtain a new binarized image including multiple first pixels, where the first pixels belong to the detail portion of the image.

[0117] Based on the first pixel and the preset pixel block side length, the detail part in the new binarized image is divided into blocks to obtain multiple detail pixel blocks;

[0118] The super-resolution reconstruction of each detail pixel block is performed by using an iterative back projection method to obtain the super-resolution pixel value of each pixel in the detail pixel block.

[0119] Furthermore, the device also includes a sixth processing unit, which is used for:

[0120] The original image is super-resolution magnified using an interpolation method to obtain a magnified interpolated image;

[0121] The pixel values ​​in the magnified interpolated image are replaced based on the super-resolution pixel values ​​of the pixel to be determined and the super-resolution pixel values ​​of each pixel in the detail pixel block to generate the target super-resolution image.

[0122] It should be noted that those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the apparatus and each unit described above can be referred to the corresponding process in the aforementioned embodiment of the image super-resolution edge processing method, and will not be repeated here.

[0123] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements all or part of the steps of the aforementioned image super-resolution edge processing method.

[0124] The embodiments of this application can implement all or part of the aforementioned processes, or they can be accomplished by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various methods described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, ROM (Read-Only memory), RAM (Random Access Memory), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0125] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, servers, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0126] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0127] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0128] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for image super-resolution edge processing, characterized in that, Includes the following steps: The derivative matrix is ​​obtained by performing derivative calculations on the low-resolution grayscale image. Based on the non-zero response characteristic of the derivative values ​​of pixels in the derivative matrix, the edges of low-resolution grayscale images are segmented to obtain multiple edge pixel segments and their corresponding edge directions. Determine whether a target pixel in a low-resolution grayscale image is located within any edge pixel segment, wherein the target pixel has a corresponding relationship with the pixel to be determined in the super-resolution image; If so, the super-resolution pixel value of the pixel to be determined is calculated according to the target interpolation formula corresponding to the edge direction of the edge pixel segment where the target pixel is located; The edge pixel segment includes a hard edge pixel segment and a soft edge pixel segment. The non-zero derivative response characteristic of the hard edge pixel segment is that both the first and second orders have the same brief non-zero response, and the second order has both positive and negative values. The non-zero derivative response characteristic of the soft edge pixel segment is that the first order has a continuous non-zero response, and the second order non-zero response only appears at the beginning and end of the soft edge pixel segment. The edge pixel segment where the target pixel is located is the target hard edge pixel segment. The step of calculating the super-resolution pixel value of the pixel to be determined according to the target interpolation calculation formula corresponding to the edge direction of the edge pixel segment where the target pixel is located includes: Construct the equation of the edge direction line based on the coordinates of the target pixel and the edge direction corresponding to the target hard edge pixel segment; The super-resolution pixel value of the pixel to be determined is calculated based on the equation of the straight line in the edge direction and the coordinates of the nearest integer target point to the target pixel. The edge pixel segment where the target pixel is located is the target soft edge pixel segment. The step of calculating the super-resolution pixel value of the pixel to be determined according to the target interpolation calculation formula corresponding to the edge direction of the edge pixel segment where the target pixel is located includes: Construct the equation of the normal line based on the coordinates of the target pixel and the edge direction corresponding to the target soft edge pixel segment; The super-resolution pixel value of the pixel to be determined is calculated based on the equation of the normal line and the coordinates of the two nearest integer points to the target pixel.

2. The image super-resolution edge processing method as described in claim 1, characterized in that, The step of calculating the derivative of a low-resolution grayscale image to obtain a derivative matrix includes: The first-order derivative matrix and the second-order derivative matrix are obtained by calculating the derivatives of the low-resolution grayscale image using the first-order and second-order derivative operators, respectively.

3. The image super-resolution edge processing method as described in claim 1, characterized in that, The edge direction is determined based on the coordinates of the first and last pixels in the edge pixel segment, and the formula for calculating the edge direction value is as follows: tanθ = (b2-b1) / (a2-a1) In the formula, tanθ represents the edge direction value, (a1,b1) represents the coordinates of the first pixel, and (a2,b2) represents the coordinates of the last pixel.

4. The image super-resolution edge processing method as described in claim 1, characterized in that, After the step of calculating the super-resolution pixel value of the pixel to be determined according to the target interpolation calculation formula corresponding to the edge direction of the edge pixel segment where the target pixel is located, the method further includes: Edge detection is performed on the original image corresponding to the low-resolution grayscale image using an edge detection operator to obtain a binarized image. The edge portion of the binarized image is processed based on the coordinates of each pixel in each edge pixel segment to obtain a new binarized image including multiple first pixels, where the first pixels belong to the detail portion of the image. Based on the first pixel and the preset pixel block side length, the detail part in the new binarized image is divided into blocks to obtain multiple detail pixel blocks; The super-resolution reconstruction of each detail pixel block is performed by using an iterative back-projection method to obtain the super-resolution pixel value of each pixel in the detail pixel block.

5. The image super-resolution edge processing method as described in claim 4, characterized in that, After the step of performing super-resolution reconstruction processing on each detail pixel block using an iterative back-projection method to obtain the super-resolution pixel value of each pixel in the detail pixel block, the method further includes: The original image is super-resolution magnified using an interpolation method to obtain a magnified interpolated image; The pixel values ​​in the magnified interpolated image are replaced based on the super-resolution pixel values ​​of the pixel to be determined and the super-resolution pixel values ​​of each pixel in the detail pixel block to generate the target super-resolution image.

6. An image super-resolution edge processing device, characterized in that, include: The first processing unit is used to perform derivative calculations on the low-resolution grayscale image to obtain the derivative matrix; The second processing unit is used to segment the edges of the low-resolution grayscale image based on the non-zero response characteristics of the derivative values ​​of the pixels in the derivative matrix, and to obtain multiple edge pixel segments and their corresponding edge directions. The third processing unit is used to determine whether the target pixel in the low-resolution grayscale image is located within any edge pixel segment, and the target pixel has a corresponding relationship with the pixel to be determined in the super-resolution image. The fourth processing unit is used to calculate the super-resolution pixel value of the pixel to be determined according to the target interpolation calculation formula corresponding to the edge direction of the edge pixel segment where the target pixel is located, if the condition is met. The edge pixel segment includes a hard edge pixel segment and a soft edge pixel segment. The non-zero derivative response characteristic of the hard edge pixel segment is that both the first and second orders have the same brief non-zero response, and the second order has both positive and negative values. The non-zero derivative response characteristic of the soft edge pixel segment is that the first order has a continuous non-zero response, and the second order non-zero response only appears at the beginning and end of the soft edge pixel segment. The edge pixel segment where the target pixel point is located is the target hard edge pixel segment, and the fourth processing unit is specifically used for: Construct the equation of the edge direction line based on the coordinates of the target pixel and the edge direction corresponding to the target hard edge pixel segment; The super-resolution pixel value of the pixel to be determined is calculated based on the equation of the straight line in the edge direction and the coordinates of the nearest integer target point to the target pixel. The edge pixel segment where the target pixel point is located is the target soft edge pixel segment, and the fourth processing unit is further used for: Construct the equation of the normal line based on the coordinates of the target pixel and the edge direction corresponding to the target soft edge pixel segment; The super-resolution pixel value of the pixel to be determined is calculated based on the equation of the normal line and the coordinates of the two nearest integer points to the target pixel.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions that, when executed by a computer, cause the computer to perform the image super-resolution edge processing method according to any one of claims 1 to 5.

Images