Super-resolution image reconstruction method, device, electronic device and storage medium

Abstract

The present invention discloses a super-resolution image reconstruction method, device, electronic device and storage medium, the method comprising: obtaining a pre-trained filter; sequentially selecting a pixel subset of size k*k centered on each pixel point in a low-resolution image to be reconstructed, and calculating the attribute value of each pixel subset; obtaining a corresponding filter parameter from the filter according to the attribute value of each pixel subset; obtaining a super-resolution image according to each pixel subset and the corresponding filter parameter, the filter is obtained after training based on a pre-acquired low-resolution sample image and a high-resolution sample image describing the same content, and the filter parameter is used to characterize the mapping relationship between the pixel subset of size k*k centered on each feature pixel point in the low-resolution sample image and the corresponding target pixel point in the high-resolution sample image. The present invention can reduce the amount of calculation and storage resources in the process of super-resolution image reconstruction, and the reconstruction speed is faster and the cost is lower.

Description

Technical Field

[0001] The present invention relates to the technical field of super-resolution image reconstruction, and in particular to a super-resolution image reconstruction method, device, electronic equipment and storage medium. Background Art

[0002] High-resolution images have higher pixel density, clearer picture quality and richer detail information. Restoring low-resolution images into high-resolution images with rich detail information can not only improve people's visual experience but also facilitate the completion of subsequent image processing tasks.

[0003] Image super-resolution, or super-resolution for short, usually refers to the enlargement of resolution, such as enlarging a 512x512 image to a 1024x1024 image. The zoom level scale is often used to describe the relationship between the image size before and after super-resolution, which refers to how many times the length and width of the image are enlarged. For example, when a 512x512 image is enlarged to a 1024x1024 image, the magnification is 2. Correspondingly, super-resolution processing is a method of converting a low-resolution image into a high-resolution image through interpolation or reconstruction. Interpolation methods generally include nearest neighbor interpolation, linear interpolation, bilinear interpolation, and bicubic interpolation. These interpolation methods are based on one or more pixels of the original image for each pixel in the super-resolution image. They are calculated in a fixed way, based on the corresponding relationship between resolution points in the image, to improve the resolution of the image. For example, the most commonly used bicubic interpolation algorithm generates one pixel in the super-resolution image based on the 4×4 pixels in the original image with different weights, such as Figure 1 As shown in the figure, this method uses a fixed calculation method for each generated pixel and ignores the color texture and other features in the original image area. The generated image quality is not high, but its advantage is that the amount of calculation is low.

[0004] In order to solve the problem of low image quality when interpolation methods are used to reconstruct low-resolution images, learning-based image reconstruction methods have gradually been developed to perform super-resolution operations on images instead of interpolation methods. However, existing learning-based image reconstruction methods require the image to be interpolated and amplified before performing corresponding calculations, which requires more storage space and computing power, resulting in excessively high hardware costs in actual applications.

[0005] Therefore, it is necessary to provide an improved technical solution to overcome the above technical problems existing in the prior art. Summary of the invention

[0006] In order to solve the above technical problems, the present invention provides a super-resolution image reconstruction method, device, electronic device and storage medium, which can reduce the amount of calculation and storage resources in the super-resolution image reconstruction process, and achieve faster reconstruction speed and lower cost.

[0007] According to a first aspect of the present disclosure, a super-resolution image reconstruction method is provided, comprising: obtaining a pre-trained filter; sequentially selecting pixel subsets of size k*k centered on each pixel point in a low-resolution image to be reconstructed, and calculating the attribute value of each pixel subset in the low-resolution image to be reconstructed; obtaining corresponding filter parameters from the filter according to the attribute value of each pixel subset in the low-resolution image to be reconstructed; obtaining a super-resolution image of the low-resolution image to be reconstructed according to each pixel subset in the low-resolution image to be reconstructed and the corresponding filter parameters, wherein the filter is obtained after training based on pre-acquired low-resolution sample images and high-resolution sample images describing the same content, and the filter parameters in the filter are used to characterize the mapping relationship between the pixel subset of size k*k centered on each feature pixel point in the low-resolution sample image and the corresponding target pixel point in the high-resolution sample image; and k is a natural number greater than 1.

[0008] Optionally, the method for obtaining the filter by training based on pre-acquired low-resolution sample images and high-resolution sample images describing the same content includes: sequentially selecting pixel subsets of size k*k centered on each feature pixel point in the low-resolution sample image, and calculating the attribute value of each pixel subset in the low-resolution sample image; grouping the attribute values ​​of each pixel subset in the low-resolution sample image; obtaining target pixel points corresponding to each pixel subset in the low-resolution sample image from the high-resolution sample image; sequentially calculating the mapping matrix between each pixel subset in the low-resolution sample image and the corresponding target pixel points; and obtaining the filter parameters corresponding to each grouping according to the grouping of each pixel subset in the low-resolution sample image and the calculated mapping matrix.

[0009] Optionally, when selecting a pixel subset of size k*k with a pixel point of the image as the center, the method further includes: performing Filling of pixels in rows or columns, wherein if the straight line where the edge row or column pixels of the image are located is taken as the axis of symmetry, then in the image after pixel filling, the pixel values ​​of two pixels that are axially symmetrical to each other are the same.

[0010] Optionally, the attribute value of each pixel subset includes: an angle index, a correlation index and an intensity index; and each group corresponds to an interval of the angle index, the correlation index and the intensity index respectively.

[0011] Optionally, obtaining corresponding filter parameters from the filter according to the attribute value of each pixel subset in the low-resolution image to be reconstructed includes: obtaining grouping information of each pixel subset according to the attribute value of each pixel subset in the low-resolution image to be reconstructed; and obtaining corresponding filter parameters according to the grouping information of each pixel subset.

[0012] Optionally, obtaining a super-resolution image of the low-resolution image to be reconstructed according to each pixel subset in the low-resolution image to be reconstructed and the corresponding filter parameters includes performing the following operations on each pixel subset in the low-resolution image to be reconstructed: rotating the pixel subset in a clockwise direction at a fixed angle around its center to obtain a plurality of pixel matrices, and calculating a weighted sum of the pixel values ​​of each pixel in the pixel subset according to each pixel matrix and the corresponding filter parameter, to obtain the pixel values ​​of a plurality of pixel points corresponding to the pixel subset in the super-resolution image; or rotating the filter parameters corresponding to the pixel subset in a counterclockwise direction at a fixed angle around the center of the filter parameters to obtain a plurality of parameter matrices, and calculating a weighted sum of the pixel values ​​of each pixel in the pixel subset according to each parameter matrix and the corresponding pixel subset, to obtain the pixel values ​​of a plurality of pixel points corresponding to the pixel subset in the super-resolution image, wherein the filter parameters are in the form of a k*k matrix, used to characterize the weight of the pixel value of each pixel in the corresponding pixel subset.

[0013] Optionally, the fixed angles include: 0 degrees, 90 degrees, 180 degrees and 270 degrees.

[0014] Optionally, k is an odd number.

[0015] Optionally, the value of k includes any one of 3, 5, 7, 9, and 11.

[0016] According to a second aspect of the present disclosure, a super-resolution image reconstruction device is provided, comprising: a pixel subset acquisition unit, used to sequentially select pixel subsets of size k*k centered on each pixel point in a low-resolution image to be reconstructed; a first calculation unit, used to calculate the attribute value of each pixel subset in the low-resolution image to be reconstructed; an image reconstruction unit, used to obtain a super-resolution image of the low-resolution image to be reconstructed according to a pre-trained filter and the attribute value of each pixel subset in the low-resolution image to be reconstructed calculated by the first calculation unit, wherein the filter is obtained after training based on pre-acquired low-resolution sample images and high-resolution sample images describing the same content, and the filter parameters in the filter are used to characterize the mapping relationship between the pixel subset of size k*k centered on each feature pixel point in the low-resolution sample image and the corresponding target pixel point in the high-resolution sample image; and k is a natural number greater than 1.

[0017] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program implements the steps in the super-resolution image reconstruction method as described above when executed by the processor.

[0018] According to a fourth aspect of the present disclosure, a storage medium is provided, comprising: a computer program or instructions are stored on the storage medium, and when the computer program or instructions are executed by a processor, the steps in the super-resolution image reconstruction method as described above are implemented.

[0019] The technical solution of the present invention can effectively reduce the amount of calculation and storage resources in the super-resolution image reconstruction process, and the reconstruction speed is faster and the cost is lower.

[0020] It should be noted that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram showing the weight relationship between pixels in the bicubic interpolation algorithm;

[0022] Figure 2 A schematic diagram showing the process of pixel conversion in a learning-based image reconstruction method;

[0023] Figure 3 A schematic diagram showing a flow chart of a super-resolution image reconstruction method provided according to an embodiment of the present invention;

[0024] Figure 4 A schematic diagram showing a flow chart of a filter training method provided according to an embodiment of the present invention;

[0025] Figure 5a A schematic diagram showing the structure of a low-resolution image provided according to an embodiment of the present invention is shown;

[0026] Figure 5b A schematic diagram showing the principle of a certain pixel subset in a low-resolution image converted by a filter according to an embodiment of the present invention;

[0027] Figure 5c A schematic diagram showing the structure of a reconstructed super-resolution image provided according to an embodiment of the present invention;

[0028] Figure 6 A schematic diagram showing the structure of a pixel subset of edge pixels in a low-resolution image provided by an embodiment of the present invention;

[0029] Figure 7a-7cSchematic diagrams showing the number of filters required at magnifications of 2 times, 3 times and 4 times according to embodiments of the present invention respectively;

[0030] Figure 8 A schematic diagram showing the structure of a super-resolution image reconstruction device provided in an embodiment of the present invention is shown;

[0031] Fig. 9 A schematic structural diagram of an electronic device provided according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0032] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. The preferred embodiments of the present invention are shown in the drawings. However, the present invention can be implemented in different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0033] Taking the RAISR image reconstruction algorithm as an example, the existing learning-based image reconstruction method includes a learning phase and an image reconstruction phase. Among them, the operation steps of the RAISR image reconstruction algorithm in the learning phase include: receiving a low-resolution image and a high-resolution image depicting the same content; appropriately compressing the low-resolution image and sharpening the high-resolution image; enlarging the low-resolution image to the size of the high-resolution image by a traditional interpolation method to obtain an interpolated enlarged image; dividing a plurality of corresponding pixel subsets from the interpolated enlarged image and the high-resolution image; calculating the attribute values ​​of the pixel subsets of the interpolated enlarged image and grouping them accordingly; for each group of the grouped pixel subsets, the corresponding filter parameters are calculated from the corresponding interpolated enlarged pixel subset and the high-resolution pixel subset. The operation steps of the RAISR image reconstruction algorithm in the image reconstruction stage include: receiving a low-resolution image; enlarging the low-resolution image by a traditional interpolation method to obtain an interpolated enlarged image; dividing a plurality of corresponding pixel subsets from the interpolated enlarged image; calculating the attribute values ​​of the pixel subsets of the interpolated enlarged image and grouping them accordingly; for each group of the grouped pixel subsets, filtering using a specific parameter filter generated in the above learning process to generate corresponding new pixels; extracting pixels from the pixel subsets of the interpolated enlarged image and the new pixel subsets generated by filtering to form a super-resolution image.

[0034] refer to Figure 2, the learning-based image reconstruction method usually trains filter parameters with a certain size (e.g., 2a, 2b, 2c) in advance, and after one-to-one data processing of the trained filter parameters with the pixel subsets of the same size selected from the low-resolution image (e.g., 1a, 1b, 1c), a high-resolution pixel (e.g., 3a, 3b, 3c) in the corresponding high-resolution image can be obtained. However, it is understandable that in the existing RAISR image reconstruction algorithm, it is first necessary to perform an operation process of interpolating and amplifying the low-resolution image to be reconstructed, and secondly, it is necessary to use a larger size or a larger number of filters to perform the conversion operation of calculating the pixel subset (because the image size of the low-resolution image increases after interpolation and amplification, and the number of pixels expressing the pixel subset characteristics is greater, which will undoubtedly increase the number of pixels contained in each pixel subset when dividing the pixel subset, or increase the number of divided pixel subsets). The whole process requires greater storage resources and computing resources to meet a large number of computing requirements, which will lead to excessively high hardware costs in actual applications.

[0035] Based on this, the present invention discloses a new super-resolution image reconstruction method. This solution processes the original low-resolution image to be reconstructed, and adopts the idea of ​​directly mapping the original low-resolution image to the high-resolution image (i.e., super-resolution image), which saves the amount of calculation and storage resources in the subsequent processing steps, and helps to reduce hardware costs. Figure 3 As shown, the method includes executing steps S1 to S4:

[0036] Specifically, in step S1, a pre-trained filter is obtained.

[0037] In this embodiment, the filter is obtained after training based on pre-acquired low-resolution sample images and high-resolution sample images describing the same content, and the filter parameters in the filter are used to characterize the mapping relationship between a pixel subset of size k*k centered on each feature pixel in the low-resolution sample image and the corresponding target pixel in the high-resolution sample image. In addition, in the present invention, k is a natural number greater than 1. Specifically, if Figure 4 As shown, in this embodiment, the method for training and obtaining a filter based on a low-resolution sample image and a high-resolution sample image that describe the same content obtained in advance includes executing steps S11 to S15:

[0038] In step S11, pixel subsets of size k*k centered on each feature pixel point in the low-resolution sample image are selected in sequence, and the attribute value of each pixel subset in the low-resolution sample image is calculated.

[0039] The training process of the filter is to learn and obtain the filter parameters in the filter, that is, the reconstruction process from the low-resolution image to the high-resolution image, that is, the filter weights in the pixel conversion, based on the corresponding multiple pixels in the pre-acquired low-resolution sample image and the high-resolution sample image that describe the same content. In this embodiment, in order to directly collect pixel subsets in the original image of the low-resolution sample image, specifically, such a pixel subset collection operation is performed for each feature pixel in the low-resolution sample image: that is, a pixel subset of size k*k centered on the feature pixel is collected. Among them, in order to ensure that the selected pixel subset can measure the graphic features of the area corresponding to the corresponding feature pixel in the low-resolution sample image, in this embodiment, the value of k is an odd number to ensure that the feature pixel is the center point of its pixel subset. At the same time, in this embodiment, the value of k should not be too large to avoid feature mixing and large calculation amount, nor should it be too small to avoid too weak features. Exemplarily, the value of k in this embodiment includes any one of 3, 5, 7, 9, and 11, such as Figure 5a As shown, in one embodiment of the present invention, k is equal to 5 to exemplarily demonstrate the technical solution of the present invention. Of course, other suitable k values ​​can also be selected for different application scenarios in practice. And it should be noted that the k value used in the training and image reconstruction process of the present invention should be consistent.

[0040] Furthermore, when the characteristic pixel points in the selected low-resolution sample image are located at the edge of the image, the pixel filling method in step S2 described below may be referred to for processing.

[0041] It can be understood that since the image size of the original low-resolution sample image is relatively small, when the size of each pixel subset to be obtained is fixed, the number of pixel subsets obtained when traversing each pixel in the low-resolution sample image will be reduced accordingly. Therefore, the amount of calculation required for the subsequent filter training, i.e., the learning process, can be greatly reduced. At the same time, based on the learning process, the image quality of the super-resolution image obtained after the subsequent reconstruction of the low-resolution image can also be improved.

[0042] Furthermore, for each pixel subset obtained, the attribute value of the pixel subset is calculated so as to group the pixel subset. Exemplarily, the attribute value of each pixel subset includes an angle index, a coherence index, and a strength index. Specifically, the three parameters of the gradient angle, gradient coherence, and gradient strength of the pixel subset can be calculated according to the definition of the gradient in the pixel subset and the local structure tensor, and then the three parameters are quantified to obtain the attribute value of the pixel subset. Among them, the definition of the local structure tensor is as follows:

[0043]

[0044] Among them, T σ Represents the local structure tensor of a subset of pixels, g x Represents the horizontal gradient of the pixel, g y Represents the vertical gradient of the pixel, T 11 , T 12 , T 21 and T 22 Respectively represent the calculation results, G σ represents the 2D Gaussian function of the pixel corresponding position under a specific standard deviation σ, where the center of the Gaussian function is the center of the pixel subset, and the size of the Gaussian kernel template is k*k. From the above formula (1), it can be concluded that the matrix T σ In fact, it is the weighted average of the T matrix of each pixel in the pixel subset, where the weight function is the Gaussian function G σ . Then, we calculate the matrix T σ After that, we can use the traditional calculation method according to the matrix T σ T 11 , T 12 , T 21 and T 22 The attribute values ​​of the pixel subset are respectively calculated. Exemplarily, the attribute values ​​of the pixel subset can be calculated using the following method:

[0045] First, according to the matrix T σ T 11 , T 12 , T 21 and T 22 Calculate and obtain multiple intermediate parameters separately:

[0046]

[0047] L1=0.5*(T 22 +T 11+D).........(3),

[0048] L2=0.5*(T 22 +T 11 -D).........(4).

[0049] Next, the eigenvector (vx,vy) corresponding to the larger eigenvalue of the corresponding pixel subset is calculated:

[0050] vx=2*T 12 .........................(5),

[0051] vy=T 22 -T 11 +D........................(6).

[0052] Afterwards, the attribute value of the pixel subset is calculated based on the feature vector (vx, vy) and the intermediate parameters:

[0053] angle index=atan2(vy,vx)........................(7),

[0054]

[0055]

[0056] It can be understood that the above method of calculating the attribute value of the pixel subset according to formula (1) to formula (9) is only an exemplary embodiment of the present invention. In other embodiments of the present invention, other conventional calculation methods can also be used to calculate the attribute value of the pixel subset, and the present invention is not limited to this.

[0057] In step S12, the attribute values ​​of each pixel subset in the low-resolution sample image are grouped.

[0058] In this embodiment, the attribute value of each pixel subset includes an angle index, a correlation index and an intensity index, and then the angle index, the correlation index and the intensity index can be divided into intervals according to the preset interval length in advance, and then the intervals of each index after the division are combined to obtain multiple groups. For example, the angle index can be divided into 12 intervals according to the preset interval length, the correlation index can be divided into 3 intervals according to the preset interval length, and the intensity index can be divided into 3 intervals according to the preset interval length. Then, after the interval combination, the attribute value of the pixel subset has 3*3*12 and a total of 108 groups. Among them, each group corresponds to an interval of the angle index, the correlation index and the intensity index respectively, and then according to the interval where the specific values ​​of the angle index, the correlation index and the intensity index in the attribute value of each pixel subset in the low-resolution sample image are calculated, the grouping information corresponding to the pixel subset can be determined. The grouping method is simple, easy to operate, and has higher accuracy.

[0059] In step S13, target pixel points corresponding to each pixel subset in the low-resolution sample image are obtained from the high-resolution sample image.

[0060] When the position coordinates of the feature pixel point corresponding to the central pixel of the pixel subset in the low-resolution sample image are (x, y), the position coordinates of the target pixel point corresponding to the pixel subset obtained in the high-resolution sample image are (n*x, n*y), where n is the magnification from the low-resolution sample image to the high-resolution sample image, and x and y are both natural numbers.

[0061] It should be noted that the positions of the coordinate axes in the high-resolution sample image and the low-resolution sample image defined in the embodiment of the present invention are the same.

[0062] In step S14, a mapping matrix between each pixel subset in the low-resolution sample image and the corresponding target pixel point is calculated in sequence.

[0063] In this embodiment, each pixel subset is a pixel matrix, and the obtained mapping matrix is ​​the filter parameter of the filter. The filter parameter in the form of the mapping matrix is ​​used to characterize the weight relationship between the pixel value of each pixel point in the pixel subset and the pixel value of the target pixel point. The calculation method of the mapping matrix can be obtained by solving according to conventional mathematical operations, which will not be described in detail in this article.

[0064] In step S15, filter parameters corresponding to each group are obtained according to the grouping condition of each pixel subset in the low-resolution sample image and the calculated mapping matrix.

[0065] In this embodiment, after obtaining the grouping information of each pixel subset in step S12 and obtaining the mapping matrix corresponding to each pixel subset in step S14, the mapping matrix corresponding to each group can be determined through the correspondence between the grouping information and the mapping matrix, and then the mapping matrix corresponding to each group can be used as the filter parameter corresponding to each group.

[0066] It should be noted that the number of low-resolution sample images and high-resolution sample images describing the same content used in the training of the filter in the embodiment of the present invention can be one or more, as long as the filter parameters corresponding to all groups can be learned after training. At the same time, when multiple filter parameters corresponding to the same group are obtained during the training process, the mean of the multiple filter parameters can be taken as the final parameter corresponding to the group, or a more detailed grouping can be performed so that one group corresponds to only m filter parameters, where m is a positive integer, and the value of m has the following functional relationship with the magnification n from the low-resolution image to the high-resolution image:

[0067]

[0068] That is, by adopting the technical solution in the embodiment of the present invention, only m filter parameters are needed to expand one pixel in the low-resolution image to be reconstructed to obtain n pixels in the super-resolution image. 2 pixels, which can greatly reduce the amount of calculation required in the image reconstruction process.

[0069] In step S2, pixel subsets of size k*k centered on each pixel point in the low-resolution image to be reconstructed are selected in sequence, and the attribute value of each pixel subset in the low-resolution image to be reconstructed is calculated.

[0070] In this embodiment, in order to directly collect pixel subsets in the original image of the low-resolution image to be reconstructed, specifically, such a pixel subset collection operation is performed on each pixel point in the low-resolution image to be reconstructed: that is, a pixel subset of size k*k centered on the pixel point is collected. Among them, in order to ensure that the selected pixel subset can measure the graphic features of the area corresponding to the corresponding pixel point in the low-resolution image to be reconstructed, the value of k in this embodiment is an odd number to ensure that the pixel point is the center point of its pixel subset. At the same time, in this embodiment, the value of k should not be too large to avoid feature mixing and large calculation amount, nor should it be too small to avoid too weak features. Exemplarily, the value of k in this embodiment includes any one of 3, 5, 7, 9, and 11, such as Figure 5aAs shown, in one embodiment of the present invention, k is equal to 5 to exemplarily demonstrate the technical solution of the present invention. Of course, other suitable k values ​​can also be selected for different application scenarios in practice. And it should be noted that the k value used in the image reconstruction process of the present invention should be consistent with the k value used in the training process.

[0071] like Figure 6 As shown, in the process of obtaining the pixel subset, since the original low-resolution image to be reconstructed has its edge pixels such as P 0 The number of pixels at the center cannot meet the needs of collecting the pixel subset P of size k*k. Therefore, when collecting pixel subsets of edge pixels in the low-resolution image to be reconstructed, an out-of-bounds problem will occur, affecting the accuracy of obtaining the pixel subset. In order to avoid this situation, when selecting a pixel subset of size k*k with the pixel point of the low-resolution image to be reconstructed as the center, the present invention also includes: collecting pixel subsets outside each boundary of the low-resolution image to be reconstructed. Pixel filling of rows or columns. For example, when k = 5, Figure 6 , then two rows or columns of pixels are filled outside each boundary of the low-resolution image to be reconstructed ( Figure 6 , the pixels in the shaded part are the original pixels of the low-resolution image to be reconstructed, and the pixels outside the shaded part are the filled pixels). At this time, if the original image of the low-resolution image to be reconstructed is an i*j array image, then the image size of the low-resolution image to be reconstructed after pixel filling will become an (i+4)*(j+4) array image, thereby avoiding the cross-border problem when collecting the pixel subset P, and ensuring that each original pixel point in the low-resolution image to be reconstructed can smoothly obtain a pixel subset of size k*k centered on the original pixel point. At the same time, in one embodiment of the present invention, in order to ensure the rationality of the image reconstruction process, the pixel values ​​of the filled pixels are generated in a symmetrical manner with the edge pixels, that is, if the straight line l1 / l2 where the edge row or column pixels of the low-resolution image to be reconstructed are located is the symmetry axis, then in the low-resolution image to be reconstructed after pixel filling, the pixel values ​​of two pixels in an axisymmetric relationship are the same. Wherein, i and j are both positive integers.

[0072] It can be understood that since the image size of the original low-resolution image to be reconstructed is relatively small, that is, the number of pixels contained in the original low-resolution image to be reconstructed is relatively small, therefore, when traversing each pixel in the original low-resolution image to be reconstructed to obtain the corresponding pixel subset, the number of pixel subsets obtained will be reduced accordingly, thereby greatly reducing the amount of calculation required in the image reconstruction process. At the same time, based on the aforementioned training and learning process, the image quality of the super-resolution image obtained after the low-resolution image is reconstructed can also be improved.

[0073] Furthermore, the present invention obtains the attribute value of each pixel subset in the low-resolution image to be reconstructed, including an angle index, a correlation index and an intensity index. Exemplarily, when calculating the attribute value of each pixel subset in the low-resolution image to be reconstructed, the three parameters of the gradient angle, gradient correlation and gradient intensity of the pixel subset can be first calculated according to the definition of the gradient of the pixel subset and the local structure tensor, and then the three parameters are quantified to obtain the attribute value of the pixel subset. The definition formula of the local structure tensor is detailed in the above-mentioned formula (1) and its related description, which will not be described in detail here. And in the calculation to obtain the matrix T σ After that, the traditional calculation method can also be used according to the matrix T σ T 11 , T 12 , T 21 and T 22 The attribute value of each pixel subset in the low-resolution image to be reconstructed is calculated respectively. The specific calculation process can be understood according to the existing technology and will not be described in detail here.

[0074] In step S3, corresponding filter parameters are obtained from the filter according to the attribute value of each pixel subset in the low-resolution image to be reconstructed.

[0075] In this embodiment, step S3 further includes: obtaining grouping information of each pixel subset according to the attribute value of each pixel subset in the low-resolution image to be reconstructed; and obtaining corresponding filter parameters according to the grouping information of each pixel subset.

[0076] In step S4, a super-resolution image of the low-resolution image to be reconstructed is obtained according to each pixel subset in the low-resolution image to be reconstructed and the corresponding filter parameters.

[0077] Optionally, in a possible embodiment of the present invention, step S4 further includes performing the following operations on each pixel subset in the low-resolution image to be reconstructed: rotating the pixel subset by a fixed angle around its center to obtain multiple pixel matrices, and calculating the weighted sum of the pixel values ​​of each pixel in the pixel subset according to each pixel matrix and the corresponding filter parameter, to obtain the pixel values ​​of multiple pixel points corresponding to the pixel subset in the super-resolution image. The fixed angles include: 0 degrees, 90 degrees, 180 degrees, and 270 degrees, and the rotation process is easy to implement and does not increase the complexity of subsequent calculations.

[0078] In another possible embodiment of the present invention, step S4 further includes performing the following operations on each pixel subset in the low-resolution image to be reconstructed: obtaining a plurality of parameter matrices after rotating the filter parameters corresponding to the pixel subset by a fixed angle according to the center of the filter parameters, and calculating the weighted sum of the pixel values ​​of each pixel in the pixel subset according to each parameter matrix and the corresponding pixel subset, respectively, to obtain the pixel values ​​of a plurality of pixel points corresponding to the pixel subset in the super-resolution image. The fixed angles include: 0 degrees, 90 degrees, 180 degrees and 270 degrees, and the rotation process is easy to implement and does not increase the complexity of subsequent calculations.

[0079] Based on the above description, it can be concluded that the filter parameters in the filter obtained through training are also in the form of a k*k matrix, which is used to characterize the weight of the pixel value of each pixel in the corresponding pixel subset.

[0080] refer to Figure 5a , Figure 5b and Figure 5c For example, if the magnification factor n from the low-resolution image 10 to be reconstructed to the super-resolution image 20 is 2, then each pixel in the low-resolution image 10 to be reconstructed needs to be converted into 2*2 equal to 4 pixels after reconstruction. 0 As an example, in the reconstruction process of pixel point P 0 After obtaining a pixel subset P as the center, the grouping information of the pixel subset P is obtained by calculating the intervals corresponding to the angle index, correlation index and intensity index in the attribute value of the pixel subset P (for example, the attribute value of the pixel subset can be divided into 108 groups a1 to a108, and the 108 groups correspond to filters F1 to F108 respectively), and then according to the obtained grouping information of the pixel subset P, the corresponding filter can be selected from filters F1 to F108 for conversion. In the conversion process, Figure 5bA schematic diagram is shown in which the pixel matrix of the pixel subset P is rotated by 0, 90, 180 and 270 degrees in a clockwise direction, respectively, and then the pixel points P1, P2, P3 and P4 in the super-resolution image 20 are obtained by operating with the corresponding filters. However, it can be understood that the pixel matrix of the pixel subset P can also be kept unchanged, and only the parameter matrix of the filter corresponding to the grouping signal of the pixel subset P is rotated by 0, 90, 180 and 270 degrees in a counterclockwise direction or by 0, 270, 180 and 90 degrees in a clockwise direction, respectively, and then the pixel points P1, P2, P3 and P4 in the super-resolution image 20 are obtained by operating with the pixel subset P. Based on the same method, after traversing all the pixel points in the low-resolution image 10 to be reconstructed, multiple groups of reconstructed pixel points P1, P2, P3 and P4 can be obtained, and then all the newly generated pixels can be combined according to a predetermined rule to obtain the final reconstructed super-resolution image, such as Figure 5c As shown. For example, the pixel value of a certain pixel point (denoted as Pxy) in the low-resolution image to be reconstructed is calculated to obtain Figure 5c Taking the pixel values ​​of the four pixels P1, P2, P3 and P4 in the dashed box in the super-resolution image shown in FIG as an example, it is assumed that the pixel matrix A of the pixel subset (taking k = 3 as an example) of the centroid of the pixel point Pxy is The corresponding filter parameter matrix B is Then the pixel values ​​of the pixels P1, P2, P3 and P4 are the weighted sums of the corresponding elements in the pixel matrix A and its multiple rotation matrices and the parameter matrix B, that is:

[0081] P1=A11*B11+A12*B12+A13*B13+A21*B21+A22*B22+A23*B23+A31*B31+A32*B32+A33*B33.............(11),

[0082] P2=A31*B11+A21*B12+A11*B13+A32*B21+A22*B22+A12*B23+A33*B31+A23*B32+A13*B33.............(12),

[0083] P3=A33*B11+A32*B12+A31*B13+A23*B21+A22*B22+A21*B23+A13*B31+A12*B32+A11*B33.............(13),

[0084] P4=A13*B11+A23*B12+A33*B13+A12*B21+A22*B22+A32*B23+A11*B31+A32*B32+A31*B33.............(14).

[0085] It can be understood that the traditional bilinear interpolation method is based on the interpolation pixel falling back to the four pixels around the coordinate area of ​​the original image through two linear solutions. Under a fixed magnification ratio n, the bilinear interpolation method interpolates pixels, which is equivalent to using n pixels based on the four pixels of the original image. 2 filters. In the technical solution of the present invention, the filter parameters for generating high-resolution pixels from low-resolution pixels are obtained based on the classification and learning of graphic features, and when classifying pixel subsets in the learning or training stage, the present invention only learns to generate pixels in a quarter of the area of ​​high-resolution pixels, and then only needs to generate high-resolution pixels in the remaining three quarters of the area by rotating the pixel subsets in a clockwise direction by 0, 90, 180 and 270 degrees or rotating the filter parameters corresponding to the pixel subsets in a counterclockwise direction by 0, 90, 180 and 270 degrees in the later image reconstruction stage. Figure 7a to Figure 7c As shown, only the filter parameters corresponding to the pixels in the dotted box are learned, and the filter parameters of the remaining pixels can be obtained by rotating the filter parameters of the pixels in the dotted box. Therefore, the number of filters required in the super-resolution image reconstruction process of the present invention is smaller, which greatly reduces the overhead of storage resources and computing resources. Figure 7a A schematic diagram showing the number of filters required when the magnification is 2 and the location of the filter parameters corresponding to the high-resolution pixels that need to be learned, Figure 7b A schematic diagram showing the number of filters required when the magnification is 3 and the location of the filter parameters corresponding to the high-resolution pixels that need to be learned, Figure 7c A schematic diagram showing the number of filters required when the magnification is 4 and the position of the filter parameters of the corresponding high-resolution pixels that need to be learned is shown. By analogy, a person skilled in the art can also easily obtain a schematic diagram showing the number of filters required when the magnification is greater than 4 and the position of the filter parameters of the corresponding high-resolution pixels that need to be learned.

[0086] Furthermore, the present invention also discloses a super-resolution image reconstruction device, which is used to implement the above-mentioned super-resolution image reconstruction method. Figure 8 The super-resolution image reconstruction device disclosed in the embodiment of the present invention includes: a pixel subset acquisition unit 100, a first calculation unit 200 and an image reconstruction unit 300.

[0087] The pixel subset acquisition unit 100 is used to sequentially select pixel subsets of size k*k centered at each pixel point in the low-resolution image to be reconstructed.

[0088] The first calculation unit 200 is used to calculate the attribute value of each pixel subset in the low-resolution image to be reconstructed.

[0089] The image reconstruction unit 300 is used to obtain a super-resolution image of the low-resolution image to be reconstructed according to the pre-trained filter and the attribute value of each pixel subset in the low-resolution image to be reconstructed calculated by the first calculation unit 200. The filter is obtained after training based on the pre-acquired low-resolution sample image and high-resolution sample image describing the same content, and the filter parameters in the filter are used to characterize the mapping relationship between the pixel subset of size k*k centered on each feature pixel point in the low-resolution sample image and the corresponding target pixel point in the high-resolution sample image.

[0090] In specific implementation, each module / unit in the super-resolution image reconstruction device can be implemented as an independent entity, or can be arbitrarily combined and implemented as the same or several entities. At the same time, the specific implementation of each module / unit in the super-resolution image reconstruction device described above can refer to the above-mentioned super-resolution image reconstruction method embodiment, which will not be repeated here.

[0091] The embodiment of the present invention further provides an electronic device 400, such as Fig. 9 As shown, it includes a memory 420, a processor 410, and a program stored in the memory 420 and executable on the processor 410. When the program is executed by the processor 310, each process of each embodiment of the above-mentioned super-resolution image reconstruction method can be implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0092] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling related hardware through instructions, and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present invention also provides a storage medium, on which a computer program or instruction is stored, and when the computer program or instruction is executed by a processor, each process of each embodiment of the above super-resolution image reconstruction method can be implemented. Among them, the storage medium, such as a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes.

[0093] Since the instructions stored in the storage medium can execute the steps in the super-resolution image reconstruction method provided in the embodiment of the present invention, the beneficial effects that can be achieved by the super-resolution image reconstruction method provided in the embodiment of the present invention can be achieved. For details, please refer to the previous embodiment, which will not be repeated here. The specific implementation of each of the above operations can be referred to the previous embodiment, which will not be repeated here.

[0094] In summary, 1. The technical solution of the present invention adopts the idea of ​​directly mapping the original low-resolution sample image to the high-resolution sample image. The amount of data required for the interpolation image is relatively small, which saves the calculation amount and storage resources in the subsequent processing steps.

[0095] 2. The technical solution of the present invention implicitly places the bilinear interpolation process in the learning and rotation process of the filter, saving the calculation amount of at least one filtering process.

[0096] 3. The technical solution of the present invention utilizes the rotational symmetry of the interpolation filter and the experimental induction of statistical characteristics, thereby reducing the number of required filters and greatly reducing storage resource overhead.

[0097] 4. The technical solution of the present invention is to perform image reconstruction processing on the original image of a low-resolution image. Since the number of pixel points contained in the original image of the low-resolution image to be reconstructed is relatively small, when traversing each pixel point in the original image of the low-resolution image to be reconstructed to obtain the corresponding pixel subset, the number of pixel subsets obtained will be reduced accordingly, and the size of each pixel subset obtained will also be reduced accordingly, thereby further reducing the amount of calculation in the image reconstruction process.

[0098] Finally, it should be noted that: Obviously, the above embodiments are only examples for clearly explaining the present invention, and are not intended to limit the implementation methods. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from this are still within the scope of protection of the present invention.

Claims

1. A super-resolution image reconstruction method, characterized in that: include: Get pre-trained filters; Selecting pixel subsets of size k*k centered on each pixel point in the low-resolution image to be reconstructed in sequence, and calculating the attribute value of each pixel subset in the low-resolution image to be reconstructed, wherein the attribute value of each pixel subset includes: an angle index, a correlation index and an intensity index; Obtaining corresponding filter parameters from the filter according to the attribute value of each pixel subset in the low-resolution image to be reconstructed; Obtaining a super-resolution image of the low-resolution image to be reconstructed according to each pixel subset in the low-resolution image to be reconstructed and the corresponding filter parameters, The filter is obtained after training based on pre-acquired low-resolution sample images and high-resolution sample images describing the same content, and the filter parameters in the filter are used to characterize the mapping relationship between a pixel subset of size k*k centered on each feature pixel point in the low-resolution sample image and a corresponding target pixel point in the high-resolution sample image; and k is a natural number greater than 1; Calculating the attribute value of each pixel subset in the low-resolution image to be reconstructed, comprising: The gradient angle, gradient correlation and gradient intensity parameters of the pixel subset are calculated according to the definition of the gradient in the pixel subset and the local structure tensor; quantizing the gradient angle, the gradient correlation and the gradient intensity parameter of the pixel subset to obtain the attribute value of the pixel subset; The method for obtaining the filter by training based on the pre-acquired low-resolution sample image and high-resolution sample image describing the same content comprises: Sequentially selecting pixel subsets of size k*k centered on each feature pixel point in the low-resolution sample image, and calculating the attribute value of each pixel subset in the low-resolution sample image; Grouping the attribute values ​​of each pixel subset in the low-resolution sample image; Acquire target pixel points corresponding to each pixel subset in the low-resolution sample image from the high-resolution sample image; Calculating in sequence the mapping matrix between each pixel subset in the low-resolution sample image and the corresponding target pixel point; The filter parameters corresponding to each group are obtained according to the grouping condition of each pixel subset in the low-resolution sample image and the calculated mapping matrix.

2. The super-resolution image reconstruction method according to claim 1, characterized in that: When selecting a pixel subset of size k*k with the pixel point of the image as the center, it also includes: performing pixel padding of rows or columns, If the straight line where the edge row or column pixel points of the image are located is taken as the axis of symmetry, then in the image after pixel filling, the pixel values ​​of two pixel points that are in an axisymmetric relationship with each other are the same.

3. The super-resolution image reconstruction method according to claim 1, characterized in that: Each group corresponds to an interval of the angle index, the relevance index and the strength index respectively.

4. The super-resolution image reconstruction method according to claim 1, characterized in that: Acquiring corresponding filter parameters from the filter according to the attribute value of each pixel subset in the low-resolution image to be reconstructed includes: Acquiring grouping information of each pixel subset according to the attribute value of each pixel subset in the low-resolution image to be reconstructed; The corresponding filter parameters are obtained according to the grouping information of each pixel subset.

5. The super-resolution image reconstruction method according to claim 1, characterized in that: Obtaining a super-resolution image of the low-resolution image to be reconstructed according to each pixel subset in the low-resolution image to be reconstructed and the corresponding filter parameters includes performing the following operations on each pixel subset in the low-resolution image to be reconstructed: The pixel subset is rotated clockwise at a fixed angle around its center to obtain a plurality of pixel matrices, and a weighted sum of the pixel values ​​of each pixel in the pixel subset is calculated according to each pixel matrix and a corresponding filter parameter to obtain pixel values ​​of a plurality of pixel points corresponding to the pixel subset in the super-resolution image; or The filter parameters corresponding to the pixel subset are rotated counterclockwise by a fixed angle according to the center of the filter parameters to obtain multiple parameter matrices, and the weighted sum of the pixel values ​​of each pixel in the pixel subset is calculated according to each parameter matrix and the corresponding pixel subset to obtain the pixel values ​​of multiple pixel points corresponding to the pixel subset in the super-resolution image. The filter parameters are in the form of a k*k matrix, and are used to represent the weight of the pixel value of each pixel in the corresponding pixel subset.

6. The super-resolution image reconstruction method according to any one of claims 1 to 5, characterized in that: k is an odd number.

7. A super-resolution image reconstruction device, used to perform the super-resolution image reconstruction method according to any one of claims 1 to 6, characterized in that: The super-resolution image reconstruction device comprises: A pixel subset acquisition unit, used for sequentially selecting a pixel subset of size k*k centered at each pixel point in the low-resolution image to be reconstructed; A first calculation unit, used for calculating the attribute value of each pixel subset in the low-resolution image to be reconstructed; an image reconstruction unit, configured to obtain a super-resolution image of the low-resolution image to be reconstructed according to a pre-trained filter and the attribute value of each pixel subset in the low-resolution image to be reconstructed calculated by the first calculation unit, The filter is obtained after training based on pre-acquired low-resolution sample images and high-resolution sample images describing the same content, and the filter parameters in the filter are used to characterize the mapping relationship between a pixel subset of size k*k centered on each feature pixel point in the low-resolution sample image and the corresponding target pixel point in the high-resolution sample image; and k is a natural number greater than 1.

8. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program implements the steps of the method according to any one of claims 1 to 6 when executed by the processor.

9. A storage medium, characterized in that: The storage medium stores a computer program or instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 6.

Images