Interpolation-based transparent structured self-learning super-resolution image reconstruction method

Abstract

The application discloses an interpolation-based transparent structured self-learning super-resolution image reconstruction method, which comprises the following steps: 1, uploading a low-resolution image to be processed to a computer and storing the low-resolution image in a corresponding low-resolution input image file of the computer; 2, input image pre-processing; 3, image feature information extraction, classifying and extracting a lower-resolution image obtained by down-sampling the pre-processed image; 4, image feature information classification, classifying each 3*3 chessboard into corresponding categories, and the approximate classification mode is as follows: according to the pixel values of the nine pixels in the 3*3 chessboard, the nine pixels are divided into large-area pixels and small-area pixels; 5, calculation of a super-resolution reconstruction coefficient based on interpolation; 6, construction of a decision maker of the feature information class; and 7, reconstruction of the low-resolution image into a high-resolution image. The application can well preserve the texture and edge information of the low-resolution image, the calculation process is simple, and the quality of the high-resolution image obtained by reconstruction is better.

Description

Technical Field

[0001] This invention belongs to the field of computer digital image processing and relates to a method and apparatus for super-resolution image reconstruction. Specifically, it involves a self-learning super-resolution image reconstruction method based on interpolation and transparent structure, a structural feature classification method, and the construction of a classification decision unit. Background Technology

[0002] Image resolution refers to the ability to represent objects of different sizes, especially smaller objects. It directly reflects important characteristics such as image sharpness and quality. High-resolution images mean that more detailed information can be captured, thus enabling better problem-solving. However, many factors affect the improvement of image resolution during digital image acquisition and processing.

[0003] Image super-resolution (SR) refers to the process of recovering a high-resolution (HR) image from a low-resolution (LR) image or image sequence. HR signifies that the image has a high pixel density, providing more detail that often plays a crucial role in applications. The most direct way to obtain high-resolution images is to use high-resolution image sensors, but this is difficult to achieve in many situations and large-scale deployments due to limitations in sensor and optical device manufacturing processes and costs. Therefore, acquiring HR images using existing equipment through super-resolution technology has significant practical implications. Super-resolution reconstruction technology is currently an effective method for improving image resolution. It only requires computer processing and does not require improvements in hardware quality to obtain high-resolution images, making it very low-cost. Image super-resolution reconstruction integrates technologies from image processing, signal processing, sensors, and artificial intelligence. Furthermore, the applications of super-resolution are very broad, including video surveillance, facial recognition, target tracking, criminal investigation, satellite remote sensing, medical imaging, and scene rendering. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing an improved method: a transparent, structured, self-learning super-resolution image reconstruction method based on interpolation. This invention offers transparency similar to interpolation-based methods, but its performance is comparable to some machine learning-based methods, such as NARM, SCSR, and SRCNN.

[0005] The specific steps of the technical solution proposed in this invention are as follows:

[0006] Step 1: The user uploads the low-resolution image to be processed to the computer via a digital device (such as a mobile phone, digital camera, etc.) and saves it in the corresponding low-resolution input image file on the computer.

[0007] Step 2: Input image preprocessing, including removing isolated noise points from the input image. Increase the input sample size by rotating the original input image by 45° (e.g., ...). Figure 3 (As shown) Increase the total number of samples.

[0008] Definition of isolated noise points: such as Figure 4 As shown in (b), an isolated noise point is a pixel that satisfies the following two conditions: (1) its value is greater than (or less than) the value of its eight surrounding pixels; (2) the minimum difference between the center pixel and its surrounding pixels is greater than 10.

[0009] Method for removing isolated noise points: The middle value is replaced by the average of its eight surrounding pixels, such as... Figure 4 As shown in (b) and (c). Figure 5 (a) and (b) are the original images. Figure 5 (c) is the graph after removing isolated noise.

[0010] Step 3: Image Feature Extraction. The preprocessed image is downsampled, and then the lower-resolution image obtained after downsampling is classified and extracted. Since all categories are based on 3×3 grids (units of size 3×3 consisting of 9 pixels), each classified 3×3 grid is a sample within that category. Then, the pixels in the 3×3 grid are classified. Therefore, the image feature information of each 3×3 grid in the input image is extracted first.

[0011] Lower resolution image: obtained by downsampling the preprocessed low resolution image, such as... Figure 6 As shown, parts_1 / 2 / 3 / 4 in the figure are taken from low-resolution image I. LR The four positions

[0012] I LR (2i,2j),I LR (2i,2j+1),I LR (2i+1,2j) and I LR (2i+1,2j+1), assuming I LR If the size is 2M×2N, then i=0,1,..,M-1,j=0,1,....,N-1. For example: the lower resolution image LLR_2 is composed of the lower resolution image I. LR The middle position is I LR It consists of all the pixels in (2i, 2j+1).

[0013] Step 4: Image feature information classification. Each 3×3 chessboard grid is divided into corresponding categories. The general classification method is: based on the pixel values ​​of the 9 pixels in the 3×3 chessboard grid, they are divided into large area pixels and small area pixels.

[0014] Step 4.1: Let the 9 pixels in the 3×3 chessboard be A to I. Then sort them from largest to smallest value, resulting in a1 to a9.

[0015] Step 4.2: Calculate the difference between adjacent pixel values, resulting in 8 differences: a1-a2, a2-a3, a3-a4, a4-a5, a5-a6, a6-a7, a7-a8, a8-a9.

[0016] Step 4.3: Find the largest difference among the 8 differences. If the largest difference is unique and its value is 'a', then... i -a i+1 Then a1~a i It belongs to the large pixel area, and a i+1 ~a9 belongs to the small cell pixels, where i is a positive integer less than or equal to 8. For example, the largest difference is a4-a5. Then a1~a4 belong to the large cell pixels, while a5~a9 belong to the small cell pixels. The same applies to other cases. If there are two or more largest differences, then these 9 pixels are considered indistinguishable and belong to the same region pixels, that is, neither belonging to the large cell nor the small cell (this kind of undifferentiated belonging to the same region is a separate type).

[0017] Step 4.4: If the difference between the largest and smallest pixel values ​​among the 9 pixels is less than 15, then these 9 pixels are considered indistinguishable and belong to the same region.

[0018] Step 4.5: Based on the classification rules in steps 4.1 to 4.4, each pixel in the 3×3 chessboard can be identified as belonging to a specific region (large region, small region, or same region). Thus, each 3×3 chessboard grid has its own structural features, which are then defined as different feature information classes.

[0019] Step 5: Calculation of super-resolution reconstruction coefficients based on interpolation.

[0020] Step 5.1: Based on Step 4, assign all extracted 3×3 checkerboard squares to their corresponding categories. Each category will then have several sample data points. Each 3×3 checkerboard square requires inserting one pixel between two pixels (12 pixels) and one pixel at the center of four pixels (4 pixels), for a total of 16 pixels. Therefore, 16 pixels need to be inserted. So, to transform a 3×3 checkerboard square into a 5×5 square, 16 pixels need to be inserted. Figure 2 As shown.

[0021] Step 5.2: Since all the 3×3 grids are extracted from even lower-resolution images obtained by downsampling from the low-resolution images, the pixels to be inserted can find their corresponding "correct" pixel values ​​in the low-resolution images. Therefore, the "correct answer" for the pixel values ​​to be inserted for each 3×3 grid can be obtained. Thus, the least squares method is used to fit and obtain a set of coefficients for each pixel in the 3×3 grid corresponding to each pixel value to be inserted. Finally, each feature information class obtained in Step 4 will have 16 sets of coefficients (because 16 pixels need to be inserted, and each pixel corresponds to a set of coefficients).

[0022] Step 6: Construct a decision maker based on feature information:

[0023] Steps 3 to 5 extract feature information classes through structural features and calculate 16 sets of coefficients corresponding to each feature information class. Since the coefficients obtained in the above three steps are the results of all samples in each feature information class, but because the 9 pixels in each 3×3 checkerboard grid are only divided into two regions or one region, such feature information classes cannot completely and accurately describe the structural features of the image. Therefore, based on the sample structural features contained in each feature information class, the feature information classes are further subdivided, resulting in more accurate results. Therefore, this step constructs a decision-maker to make a more accurate judgment on each pixel to be inserted. (The data structure used by the decision-maker in this invention is a binary tree structure, the purpose of which is to determine whether the value of the pixel to be inserted is biased towards a large region or a small region). Since it is necessary to further subdivide several sets of 3×3 samples in each category, that is, to divide all data of the nodes in the binary tree downwards into two parts each time, it is necessary to specify corresponding standards and calculation methods to clearly distinguish the two parts of samples. Then, according to the specified multiple calculation methods, the value corresponding to each calculation method for each 3×3 checkerboard grid is calculated, and then statistics are performed. Below are the specific construction steps for inserting pixels into a feature information class:

[0024] Step 6.1: In each binary tree node, the values ​​of all samples obtained by each calculation method can be classified into two categories. Find the best calculation method and the corresponding threshold that can classify all samples of the node as the standard calculation method (which can be called the attribute of the node) and the standard judgment threshold of the node.

[0025] Step 6.2: When the total number of samples in a node or the samples meet the final condition, stop dividing downwards. Then this node can be used as a leaf node and marked as a labeled node. For example: if the percentage of pixels in the large region corresponding to the samples in this node that need to be inserted exceeds a threshold (since the "correct answer" for the samples used to build the binary tree is known in advance, it is natural to calculate whether the "correct" pixel value at this position is biased towards the large region or the small region), then the label of this leaf node is large region, and vice versa. If the samples in this node cannot be clearly distinguished, then the label of this node is defined as undetermined.

[0026] Step 6.3: Each extracted feature information class will have several samples. Each position in the samples that needs to have a pixel inserted has a corresponding "correct answer," thus determining whether the pixel at each position is biased towards a large or small area. Therefore, each position can be divided into two parts: one part of the samples biased towards a large area, and the other part of the samples biased towards a small area. Following the method in Step 5, a set of coefficients can be calculated for each part: a set of coefficients for the large area bias and a set for the small area bias. Ultimately, each position where a pixel needs to be inserted will yield three sets of coefficients: a coefficient for the large area bias, a coefficient for the small area bias, and a combined coefficient. In summary, a feature information class will ultimately have 48 sets of coefficients calculated and stored.

[0027] Step 7: Super-resolution reconstruction process. Reconstruct the low-resolution image into a high-resolution image.

[0028] Step 7.1: Use the raw, low-resolution image that has not been preprocessed as the input image.

[0029] Step 7.2: Traverse each 3×3 chessboard square in the input image and classify the 3×3 chessboard squares according to the method in Step 4 to obtain the class corresponding to the current 3×3 chessboard square.

[0030] Step 7.3: Each 3×3 grid needs to insert 16 pixels. Then, the input of the 3×3 chessboard grid is fed into the binary tree decision-maker corresponding to each position. When walking down from the root to the leaf node, the label corresponding to that node will be known.

[0031] Step 7.4: Each location where a pixel value needs to be inserted will eventually receive a label. Based on the label, the corresponding coefficient is found, and then the value at that location is calculated. By traversing the entire image, all values ​​can be calculated.

[0032] Step 7.5: For the entire image, if we iterate through all the 3×3 checkerboard squares and calculate all the interpolations, there will be multiple calculation results for each position where a pixel needs to be inserted. The processing method here is to directly take the average of all the calculated values ​​as the final pixel value at this position.

[0033] Step 7.6: After obtaining the pixel values ​​at all locations, output the final result of the obtained pixel matrix in the format of an image.

[0034] The beneficial effects of this invention are:

[0035] The core technical features of this invention are a novel method for extracting and classifying image structural features and an execution decision-maker for pixel classification. The proposed method uses only the obtained low-resolution image to learn its limited structural features through an interpolation algorithm to obtain a high-resolution image. Furthermore, this invention provides a structural feature classification method that divides all samples into several groups, mapping each class of lower-resolution images to a coefficient matrix to calculate the pixel values ​​of the low-resolution images. To increase the number of samples and the diversity of structural features, this invention expands the samples by rotating the images to enhance the sample data and obtain new images. This increases the total number of samples without introducing new structural features that are completely inconsistent with the image, thus avoiding impacting the overall result. Building upon the above, to make the results more accurate, this invention also proposes an execution decision-maker based on a binary tree structure, dividing each group into two subgroups to refine the corresponding mapping matrix. Finally, the high-resolution image is obtained from the given low-resolution image using the mapping matrix.

[0036] This invention inherits the advantages of interpolation-based methods, which are transparent to the user and achieve better performance than interpolation-based methods. Furthermore, this invention belongs to a self-learning method using a single image itself, while inheriting the advantages of learning-based methods and further improving performance by increasing sampling data.

[0037] This invention consists of a decision executor and a grid feature map. It learns on its own using a given image without requiring any additional samples, and the entire process is transparent to the user. Experimental results show that this method achieves better results compared to interpolation-based and learning-based methods. Furthermore, this invention is a structured method capable of classifying different features based on various conditions using the grid feature map. Moreover, the executor can be modified and adjusted to adapt to different feature classifications. This also means that this invention can be easily applied to other computer vision applications, such as image segmentation or edge detection. Attached Figure Description

[0038] Figure 1 This is the algorithm framework of the present invention.

[0039] Figure 2 This is a schematic diagram of image interpolation.

[0040] Figure 3 This is an image rotated 45°.

[0041] Figure 4 Methods for marking and processing isolated noise points

[0042] Figure 5 This section shows the isolated noise points in a specific image and the processed image.

[0043] Figure 6 This is a schematic diagram illustrating the process of obtaining an even lower resolution image from a low-resolution image through downsampling.

[0044] Figure 7 To list the structural features of the two categories.

[0045] Figure 8 It consists of four 3×3 chessboard squares that can be obtained by rotation.

[0046] Figure 9 Given 9 known pixels and 16 pixels to be inserted.

[0047] Figure 10 This is a classification diagram of a node in a binary tree.

[0048] Figure 11 This involves constructing a binary decision tree and the information in each node.

[0049] Figure 12 A schematic diagram illustrating the general process of obtaining the label for a single pixel in a binary tree decision maker.

[0050] Figure 13 The image shows a low-resolution image of the house and a high-resolution image after super-resolution reconstruction. Detailed Implementation

[0051] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments:

[0052] Existing technologies mainly fall into two categories: interpolation methods and learning methods. Traditional interpolation methods have limited ability to handle complex situations, so interpolation-based super-resolution image reconstruction methods may not achieve good performance in some complex cases. Learning-based methods typically require sufficient sampling for training and generally outperform interpolation-based methods. However, their performance is constrained by the training dataset, and the applicability of the dataset is often negatively correlated with accuracy; therefore, its generalization ability is a key issue. Furthermore, the learning process of learning methods is usually not transparent to the user, often referred to as the "black box problem."

[0053] Image interpolation: Image interpolation transforms a low-resolution M×N image into a high-resolution 2M×2N image by inserting pixels. For example... Figure 2 As shown, Figure 2 (a) can be viewed as a 3×3 low-resolution image, so the empty space in the middle needs to be filled with appropriate pixels to become a 5×5 high-resolution image.

[0054] This invention is a self-learning super-resolution image reconstruction method based on interpolation and transparent structure. The main implementation process consists of the following seven steps.

[0055] Step 1: Input the image to be processed. The user uploads the low-resolution image to be processed to the computer via a digital device (such as a mobile phone photo) and manually places the image to be processed into the specified folder.

[0056] Step 2: Preprocess the input low-resolution image. First, perform simple smoothing on the image. This is because the image contains isolated noise points that can affect subsequent calculation results, such as... Figure 5 As shown, these isolated noise points need to be removed beforehand; secondly, more samples need to be added. Since there is only one low-resolution image input, the number of extractable samples is small, which will have a certain impact on the results. Therefore, this invention increases the total number of samples by rotating the image to obtain a new image. Figure 3 As shown; next, the low-resolution image needs to be downsampled, such as... Figure 6 As shown, four lower-resolution images are finally obtained; finally, all the images obtained from the above preprocessing are used as input.

[0057] Step 3: Downsample the preprocessed image, and then classify and extract the lower-resolution image obtained after downsampling. Since all categories are based on a 3×3 checkerboard (a unit of 3×3 pixels consisting of 9 pixels), each classified 3×3 checkerboard grid is a sample within that category. Then, the pixels in the 3×3 checkerboard grid are classified, so each 3×3 grid in the input image is extracted first.

[0058] Step 4: Extract all 3×3 checkerboard squares from the preprocessed input image, and extract the corresponding structural features for each 3×3 square according to the set method, then classify them based on different structural features. Some classification results are shown below. Figure 7 As shown in (b) and (c). The specific classification method consists of the following five steps:

[0059] (1) As Figure 7As shown in (a), the 9 pixels in the 3×3 grid are named A, B, C, D, E, F, G, H, and I. Then, the result of sorting them from largest to smallest value is a1, a2, a3, a4, a5, a6, a7, a8, and a9.

[0060] (2) Calculate the difference between adjacent pixel values, that is, a1-a2, a2-a3, a3-a4, a4-a5, a5-a6, a6-a7, a7-a8, a8-a9. A total of 8 differences will be obtained.

[0061] (3) Find the largest difference among the 8 differences obtained in (2), for example: a4-a5. Then a1, a2, a3, and a4 belong to the large area, while a5, a6, a7, a8, and a9 belong to the small area. The same applies to other cases. If there are two or more largest differences, then these 9 pixels are considered indistinguishable and belong to the same part.

[0062] (4) There is a special case here: if the difference between the largest and smallest pixel values ​​among the 9 pixels is less than 15, then these 9 pixels are also considered to be indistinguishable and belong to the same part.

[0063] (5) According to the classification rules in (2) to (4), each pixel in the 3×3 chessboard can be identified by its region (large region or small region; if no region is defined, it is treated as a separate special category). Thus, each 3×3 grid has its own structural features, and its corresponding category can be found based on these different structural features. For example... Figure 7 As shown in (b) and (c), two types of structural features are listed.

[0064] Step 5: Fit the sample data from each class to obtain the optimal coefficient matrix, which will be used for subsequent interpolation calculations. The specific calculation process is as follows:

[0065] (1) According to process two, the 3×3 grid can be divided into several categories, such as Figure 7 As shown, Figure 7 (b) and (c) respectively illustrate two of these structures. Therefore, if a pixel at a given location belongs to either a large region or a small region, then each pixel has two possibilities. So theoretically, there will be a total of 2... 9 =512 categories. However, in actual classification, classes with similar structural features can be grouped together. For example, if one class can be transformed into another class through rotation, then these two classes can be merged into the same class. Figure 8 As shown, these four categories can be grouped into one category for unified processing. That is, all samples are placed into a set for processing.

[0066] (2) After all samples have been assigned to their corresponding class sets, the mapping coefficients need to be calculated, such as... Figure 9 As shown in (a) and (b), grid-based filling is mainly completed in the following three steps. First, determine the grid based on p. i The type of 3×3 chessboard grid formed by i = 1, 2, ..., 9. Where p i This represents pixels and their corresponding pixel values. Secondly, Q is learned between the low-resolution image and the downsampled even lower-resolution image, using all samples in each class and their corresponding pixel positions. j Determined k-th type parameter j = 1, 2, ..., 16, this is achieved by minimizing the sum of squared errors for each grid sample, i.e. Where l represents the sample index of class k, and L is the total number of samples in class k. Finally, class k corresponding to the 3×3 checkerboard grid can be obtained, and the sample to be inserted at pixel position Q can be determined. j The value is set to j = 1, 2, ..., 16. Therefore, the purpose of this process is to obtain all the parameters. i = 1, 2, ..., 9, j = 1, 2, ..., 16, k = 1, 2, ..., the maximum value of k is the total number of 3×3 chessboard grids.

[0067] (3) After all the parameters are calculated, the parameter matrix corresponding to each class can be obtained.

[0068] Step Six: Construct a decision generator for each insertion pixel position in each class using all samples in each class. The data structure of the decision generator is a binary tree structure. This process mainly consists of three steps:

[0069] First, we need to define the following two concepts:

[0070] “Label”: As a sample for training the decision-maker, we already know the value of a pixel location that needs to be inserted, so we can calculate whether the point at this location is biased towards a large area or a small area, and thus we know the “label” of this location.

[0071] The sample consists of all the information of a 3×3 checkerboard grid (9 pixel values ​​and relative position information), and a "label" for the pixel to be inserted at a certain position.

[0072] (1) Seven attributes are defined, i.e., seven calculation methods. All samples in each node will be calculated into seven values. Assuming G is the sample data of a 3×3 chessboard, G l and G s These are subsets of G, namely the large area pixel set and the small area pixel set.

[0073]

[0074] The calculation formula is as follows:

[0075]

[0076]

[0077] A3:MaxxDiff(G)=max(G)-min(G) (3)

[0078]

[0079]

[0080]

[0081] A7:gradient sobel (G) (7)

[0082] In Equation 1, Average(G) is the average of all elements in matrix G. In Equation 2, StandardDeviation(G) is the standard deviation of all elements in matrix G. In Equation 3, max(G) and min(G) are the maximum and minimum elements of matrix G, respectively. In Equation 3, MaxxDiff(G) is the difference between the maximum and minimum elements of matrix G. In Equation 7, gradient... sobel (G) is the Sobel gradient of matrix G.

[0083] (2) First, compare the values ​​calculated by each calculation method for each sample. Second, take all the calculated values ​​obtained by all calculation methods for each sample as a set of data. Then, find the optimal threshold that can divide this set of data "best" (i.e., the largest TB value, see Note 2 below for details) and record the TB value. Finally, compare the TB values ​​corresponding to the optimal thresholds of the other 6 calculation methods excluding the parent node's calculation method (here it is stipulated that the calculation method selected by a node is different from the calculation method selected by the parent node). Finally, find the calculation method with the largest TB value and the optimal threshold as the storage information of the current node.

[0084]

[0085] Note 1: Equation 8 is the provided evaluation formula, such as... Figure 10This demonstrates the binary search process of a node, where D represents all samples from the parent node of N entering node N, and D1 and D2 are all samples entering nodes N1 and N2 after judging based on the conditions Ai≤thi and Ai>thi, respectively. The "labels" and in this example represent the number of samples biased towards the smaller and larger regions, respectively, j = 1, 2, and λ (set to 0.55 in this paper) is the balancing parameter. The value of λ represents the weight of the difference in the calculation of the K value; that is, the larger λ is, the greater the weight of the difference in the calculation of the K value. Therefore, the classification standard tends to distribute all samples more evenly.

[0086] Note 2: Since the decision maker's data structure is a binary tree structure, the total samples in each node need to be divided into two parts according to certain rules, which are then assigned to the left and right subtrees respectively. The specific rules are as follows: (1) The difference in the number of samples assigned to the left and right subtrees should be as small as possible. (2) In each subtree, the proportion of samples belonging to the same "label" should be as large as possible, i.e., ... Figure 10 As shown, node N is divided into two nodes, N1 and N2. For node N1, the proportion of all data D1 containing the same "label" should be as high as possible. For example, if 60% of the "label" in node N is "cell", then the total proportion of "label" in N1 should be as high as possible, possibly 80% or 90%. The same applies to N2. Finally, the two rules above can be expressed using evaluation formula 8.

[0087] (3) Finally, all nodes will record their corresponding attributes and optimal thresholds, constructing a binary tree decision maker as follows: Figure 11 As shown.

[0088] Note 3: Rules for determining whether a node is a leaf node: First, if the proportion of a certain "label" in a node is greater than or equal to 90%, then it is considered a leaf node, and the corresponding information will be stored in this node. The information is that when test data is input, if it reaches this leaf node, then the pixel is predicted to be biased towards the region to which the "label" of this record belongs. Second, if the total number of samples in a node is small, that is, less than a certain threshold X (here X is 30), then it will not continue to be divided downwards. "Label" statistics will be performed on all samples in the node. If the proportion of a certain "label" is greater than 85%, then this "label information" will be stored. If it is less than 85%, it cannot be determined, and it cannot be determined whether it is a third type of "label".

[0089] Step 7: Input the low-resolution image to be processed. Using the classification method from Step 4 and the decision-maker constructed in Step 6, calculate the pixel value for each insertion location. Finally, output the resulting super-resolution image in image format, such as... Figure 13As shown, this process mainly consists of five steps:

[0090] (1) Traverse every 3×3 grid in the original low-resolution input image, and then classify the current 3×3 chessboard grid according to the method in step four. Then find the corresponding class.

[0091] (2) Figure 9 As shown, we need to calculate the pixel value of the gray circle at each location (taking Q4 as an example), and use the decision-maker built in process four to determine which region Q4 is more biased towards. Figure 12 As shown, the topmost circle represents the input pixel Q4 to be inserted. The black circles in the middle represent non-leaf nodes, the gray circles represent leaf nodes, and the bottom circles are white, black, and gray, corresponding to the labels "large," "small," and "unknown," respectively. Initially, Q4 starts from the root node in the decision maker. The input pixel iterates through its subtree according to the calculation method and threshold recorded by the corresponding node, finally reaching the leaf node. The label of the leaf node is set to the biased result corresponding to that pixel. Here, we assume that the label obtained by Q4 is "large."

[0092] (3) Once we know the label of Q4, we can find the position of Q4 in the corresponding class and the coefficient matrix labeled "large". Then we can calculate the result |Q4| at that position and use it as the value to be inserted at the current position.

[0093] (4) Based on (2) and (3), we can calculate the pixel value corresponding to each position. However, for the entire image, if we iterate through all the 3×3 checkerboard grids and calculate all the interpolations, there will be multiple calculation results for each position where a pixel needs to be inserted. The processing method here is to directly take the average of all calculated values ​​as the final pixel value for this position.

[0094] (5) Output the final result in an image format to obtain the final super-resolution image, such as... Figure 13 As shown.

[0095] The following is a table comparing objective evaluation indicators for the super-resolution reconstruction of this invention, and a detailed description of the application effects of this invention is provided:

[0096] Table 1 Comparison of PSNR and SSIM for various methods

[0097]

[0098] Table 1 compares the average values ​​for datasets Set5 (5 images) and Set14 (14 images). Set5 and Set14 are low-complexity single-image super-resolution datasets based on non-negative neighborhood embeddings. These training sets are used for single-image super-resolution reconstruction, i.e., reconstructing high-resolution images from low-resolution images to obtain more detail. This dataset was released in 2012 by the University of Billeil and Bell Labs, France, and is currently widely used in super-resolution-related papers.

[0099] The proposed method was compared with interpolation-based methods such as Bicubic, NEDI (a novel edge-oriented interpolation method), ICBI (an interpolation method based on iterative curvature), and learning-based methods such as SCSR (a sparse representation-based SR method), NARM (a nonlocal autoregressive image interpolation method based on sparse representation), SRCNN (a convolutional neural network-based SR method), and SelfExSR (a self-similarity-driven SR algorithm). PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity) were used for evaluation; higher PSNR and SSIM values ​​indicate better method performance.

[0100] The test results show that the PSNR and SSIM scores of the method proposed in this invention are better than those of the interpolation-based method. For the learning-based NARM method, the two scores of this invention are also slightly higher.

Claims

1. A self-learning super-resolution image reconstruction method based on interpolation and transparent structuring, characterized in that... The steps include the following: Step 1: The user uploads the low-resolution image to be processed to the computer via a digital device and saves it in the corresponding low-resolution input image file on the computer; Step 2: Input image preprocessing, including removing some isolated noise points in the input image and increasing the input sample size; Step 3: Image feature extraction. The preprocessed image is downsampled, and then the lower-resolution image obtained after downsampling is classified and extracted. Step 4: Image feature information classification. Each 3×3 chessboard grid is divided into corresponding categories. The classification method is: based on the pixel values ​​of the 9 pixels in the 3×3 chessboard grid, they are divided into large area pixels and small area pixels. Step 5: Calculation of super-resolution reconstruction coefficients based on interpolation; Step 6: Construct a decision-maker based on feature information; Step 7: Super-resolution reconstruction, reconstructing the low-resolution image into a high-resolution image; Step 5 is implemented as follows: Step 5.1: Based on Step 4, classify all the feature information of the extracted 3×3 chessboard into the corresponding categories. In this way, each category will have a number of sample data. For each 3×3 chessboard, a pixel needs to be inserted between two pixels and a pixel needs to be inserted in the center of four pixels, for a total of 16 pixels. Step 5.2: Since all the 3×3 checkerboard grids are extracted from the lower-resolution images obtained by downsampling the lower-resolution images, the pixels that need to be inserted can find the corresponding "correct" pixel values ​​in the lower-resolution images. Therefore, the "correct answer" for the pixel values ​​to be inserted for each 3×3 checkerboard grid can be obtained. Thus, the least squares method is used to fit and obtain a set of coefficients for each pixel in the 3×3 checkerboard grid corresponding to each pixel value to be inserted. Finally, each feature information class obtained in Step 4 will obtain 16 sets of coefficients, with one set of coefficients for each pixel. Step 5.2 After assigning all samples to their corresponding class sets, the mapping coefficients need to be calculated. The specific calculation process is as follows: First, determine by The types of 3x3 chessboard grids formed, among which, Represents the pixel and its corresponding pixel value, i = 1, 2, ..., 9; Secondly, the algorithm learns the values ​​of all samples and their corresponding pixel positions in each class between the low-resolution image and the downsampled even lower-resolution image. Determined k-th type parameter j = 1, 2, ..., 16, this is achieved by minimizing the sum of squared errors for each grid sample, as shown in the following formula: （1） Where l represents the sample number of class k, and L is the total number of samples of class k; Finally, we obtain the class k corresponding to the 3×3 chessboard grid and insert it into the pixel position. The value is set to j = 1, 2, ..., 16; This process is to obtain parameters. i = 1, 2, ..., 9, j = 1, 2, ..., 16, k = 1, 2, ..., the maximum value of k is the total number of 3×3 chessboard grids.

2. The self-learning super-resolution image reconstruction method based on interpolation and transparent structuring according to claim 1, characterized in that... Step 3 is implemented as follows: Since all categories are based on a 3×3 checkerboard grid, each sorted 3×3 checkerboard grid is a sample within that category. Then, the pixels in the 3×3 checkerboard grid are classified. Therefore, the image feature information of each 3×3 checkerboard grid in the input image is extracted first.

3. The self-learning super-resolution image reconstruction method based on interpolation and transparent structuring according to claim 1 or 2, characterized in that... Step 4 is implemented as follows: Step 4.1: Let the 9 pixels in the 3×3 chessboard be A to I, and then sort them from largest to smallest according to their values, resulting in a1 to a9; Step 4.2: Calculate the difference between adjacent pixel values, that is, obtain 8 differences, namely: a1-a2, a2-a3, a3-a4, a4-a5, a5-a6, a6-a7, a7-a8, a8-a9; Step 4.3: Find the largest difference among the 8 differences. If the largest difference is unique and its value is 'a', then... n -a n+1 Then a1~a n It belongs to the large pixel area, and a n+1 ~a9 belongs to the cell pixel, where n is a positive integer and less than or equal to 8; if there are two or more maximum differences, then these 9 pixels are considered indistinguishable and belong to the same area pixel, that is, they belong to neither the large area nor the cell. Step 4.4: If the difference between the largest and smallest pixel values ​​among the 9 pixels is less than 15, then these 9 pixels are considered indistinguishable and belong to the same region. Step 4.5: Based on the classification rules in steps 4.1 to 4.4, each pixel in the 3×3 chessboard can be identified as belonging to a specific region; thus, each 3×3 chessboard has its own structural features, which are defined as different feature information classes based on different structural features.

4. The self-learning super-resolution image reconstruction method based on interpolation and transparent structure as described in claim 3, characterized in that... The decision-maker described in step 6 uses a binary tree structure to determine whether the value of the pixel to be inserted is biased towards the large area or the small area. The specific steps for constructing the feature information class that require inserting pixels are as follows: Step 6.1: In each binary tree node, based on the fact that all the sample values ​​obtained by each calculation method can be classified into two categories, find the best calculation method and the corresponding threshold that can classify all the samples of this node as the standard calculation method and standard judgment threshold for this node. Step 6.2: When the total number of samples in a node or the samples meet the final condition, stop dividing downwards. Then this node can be used as a leaf node and marked as a label node. If the percentage of pixels in the large area corresponding to the samples in this node that need to be inserted exceeds the threshold, then the label of this leaf node is the large area, and vice versa. If the samples in this node cannot be clearly distinguished, then the label of this node is defined as undeterminable. Step 6.3: Each extracted feature information class will have several samples. Each position in the sample has a corresponding "correct answer" for the pixel to be inserted, so as to know whether the pixel at each position is biased towards a large area or a small area. Therefore, each position can divide the sample into two parts: one part of the sample belongs to the sample that is biased towards a large area, and the other part of the sample belongs to the sample that is biased towards a small area. Based on step 5, calculate a set of coefficients for each part, a set of coefficients biased towards the large area, and a set of coefficients biased towards the small area. Ultimately, each location where a pixel needs to be inserted can obtain three sets of coefficients: coefficients biased towards the large region, coefficients biased towards the small region, and comprehensive coefficients. Therefore, a feature information class will eventually calculate and store 48 sets of coefficients.

5. The self-learning super-resolution image reconstruction method based on interpolation and transparent structuring according to claim 4, characterized in that... There are seven calculation methods described in step 6.1, as follows: For each node, all samples will generate 7 values. Assume G is the sample data of a 3×3 chessboard. and yes The subsets are the large area pixel set and the small area pixel set; （2） The calculation formula is as follows: （3） （4） （5） （6） （7） （8） （9） Among them, in equation 3 It is the average of all elements in matrix G; in Equation 4 It is the standard deviation of all elements in matrix G; in Equation 5 and These are the maximum and minimum elements of matrix G, respectively; in Equation 5 It is the difference between the largest and smallest elements in matrix G; in Equation 9 It is the Sobel gradient value of matrix G.

6. The self-learning super-resolution image reconstruction method based on interpolation and transparent structure as described in claim 5, characterized in that... Step 6.2 is as follows: First, compare the values ​​calculated by each method for each sample; Secondly, all the calculated values ​​obtained by all calculation methods for each sample are taken as a set of data; Then, find the optimal threshold to divide this set of data "best" and record the TB value. The "best" division is when the TB value is the largest. Next, compare the TB values ​​corresponding to the best thresholds for the other six calculation methods excluding the parent node calculation method; Finally, the calculation method with the largest TB value and the optimal threshold are found as the storage information for the current node; (10) Where D represents all samples from the parent node of N to node N, and D1 and D2 are based on condition A. i ≤th i and A i >th i All samples that have been judged and then enter the N1 and N2 nodes are λ, which is a balancing parameter. The larger λ is, the greater the weight of the difference in the calculation of the K value.

7. The self-learning super-resolution image reconstruction method based on interpolation and transparent structuring according to claim 6, characterized in that... Step 6.2 specifically includes rules for determining whether a node is a leaf node: First, if the proportion of a node belonging to a certain "label" is greater than or equal to 90%, then it is considered a leaf node. This node will store the corresponding information. When test data is input, if it reaches this leaf node, then it is predicted that the pixel is biased towards the region to which the "label" of this record belongs. Second, if the total number of samples in a node is less than the set threshold X, it will not be further divided, and all samples in that node will be "labeled". If the proportion of a certain "label" is greater than 85%, the "label information" will be stored. If it is less than 85%, it cannot be determined and it cannot be determined to be the third type of "label".

8. The self-learning super-resolution image reconstruction method based on interpolation and transparent structuring according to claim 7, characterized in that... The super-resolution reconstruction in step 7 is implemented as follows: Step 7.1: Use the raw, low-resolution image without preprocessing as the input image; Step 7.2: Traverse each 3×3 chessboard square in the input image, classify the 3×3 chessboard squares according to the method in Step 4, and obtain the class corresponding to the current 3×3 chessboard square; Step 7.3: Each 3×3 grid needs to insert 16 pixels, and then the input of the 3×3 chessboard grid is fed into the binary tree decision-maker corresponding to each position. When walking down from the root to the leaf node, the label corresponding to the node will be known. Step 7.4: Each location where a pixel value needs to be inserted will eventually receive a label. Based on the label, find the corresponding coefficient and then calculate the value at that location; by traversing the entire image, all values ​​can be calculated. Step 7.5: For the entire image, if we iterate through all the 3×3 checkerboard squares and calculate all the interpolations, then there will be multiple calculation results for each position where a pixel needs to be inserted. The processing method here is to directly take the average of all the calculated values ​​as the final pixel value at this position. Step 7.6: After obtaining the pixel values ​​at all locations, output the final result of the obtained pixel matrix in the format of an image.

Images