A super-resolution reconstruction imaging method

Abstract

The present invention relates to the field of image processing technology, and in particular to an imaging method for super-resolution reconstruction. The method comprises the following steps: step S1, dividing a low-resolution image into a plurality of low-resolution pixel blocks, and finding a super-resolution block with the highest similarity of each low-resolution pixel block to form an original super-resolution block; step S2, establishing a geometric model and an optical model for each low-resolution pixel block; step S3, reconstructing the low-resolution pixel block through the geometric model and the optical model, and determining the matching degree between the plurality of temporary super-resolution blocks and the original super-resolution block; step S4, assembling each temporary super-resolution block with the highest matching degree to obtain a complete super-resolution image. By establishing a correspondence between low-resolution blocks and super-resolution blocks, and using nonlinear optimization to iteratively tune the geometric optical model, real-time reconstruction of super-resolution images is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an imaging method for super-resolution reconstruction. Background Art

[0002] With the development of modern society, the requirements for image quality are constantly increasing. Super-resolution reconstruction technology, as a digital image processing and enhancement method, makes it possible to provide higher quality image services.

[0003] The patent document with publication number CN110009565A discloses a super-resolution image reconstruction method based on a lightweight network, which comprises the following steps: step 1, replacing two convolution kernels in the ResBlock (residual block) of the EDSR super-resolution reconstruction model with the designed ShuffleNet unit to obtain an improved network model, wherein the ShuffleNet unit consists of four parts: 1×1GConv (point-by-point group convolution), channel shuffle, 3×3DWConv (depth separation convolution), and 1×1GConv; step 2, performing weight parameter compression training on the improved network model in step 1, comprising the following sub-steps; step 2.1, pruning the connections whose absolute values ​​of weights in the weight matrix are less than a threshold, to obtain a pruned sparse matrix; step 2.2, quantizing the weights in the sparse matrix obtained in step 2.1, and completing the numerical sharing of all weights; step 2.3, encoding the weight data using the Huffman coding method; step 3, calculating the image to be super-resolution reconstructed through the above-mentioned trained network model to obtain a reconstructed and enlarged image.

[0004] It can be seen that the super-resolution image reconstruction method based on lightweight network has the problem that the prior art requires pre-training of the model to reconstruct the low-resolution image, resulting in the inability to achieve real-time reconstruction of the super-resolution image. Summary of the invention

[0005] To this end, the present invention provides an imaging method for super-resolution reconstruction, which is used to overcome the problem in the prior art that a model needs to be trained in advance to reconstruct a low-resolution image, resulting in the inability to achieve real-time reconstruction of a super-resolution image.

[0006] To achieve the above object, the present invention provides an imaging method for super-resolution reconstruction, comprising:

[0007] Step S1, dividing the low-resolution image into a plurality of low-resolution pixel blocks, and searching a pre-stored super-resolution block having the highest similarity to each of the low-resolution pixel blocks in a pre-established super-resolution database to form an original super-resolution block;

[0008] Step S2, establishing a geometric model and an optical model for each low-resolution pixel block;

[0009] Step S3, iteratively optimizing stretching parameters, rotation parameters, and translation parameters corresponding to the geometric model, reconstructing the low-resolution pixel block by using the optimized geometric model and the optical model to obtain a plurality of temporary super-resolution reconstruction blocks, reconstructing the plurality of temporary super-resolution reconstruction blocks by using the optical model to obtain a plurality of temporary super-resolution blocks, and determining the matching degree between the plurality of temporary super-resolution blocks and the original super-resolution blocks;

[0010] Step S4, assembling each of the temporary super-resolution blocks with the highest matching degree to obtain a complete super-resolution image.

[0011] Furthermore, the step S3 comprises:

[0012] Step S31, extracting feature points in the low-resolution pixel block and establishing a matching relationship with corresponding feature points in the original super-resolution image;

[0013] Step S32, based on the feature point matching relationship, the HSV value of the low-resolution pixel block is reconstructed using the optical model M to obtain a temporary super-resolution reconstruction block, and the temporary super-resolution reconstruction block is stretched, rotated and translated according to the geometric model T to obtain an adjusted temporary super-resolution block, wherein the HSV value is the H value, S value and V value, the H value is the hue of the color space model, the S value is the saturation of the color space model, and the V value is the lightness of the color space model;

[0014] Step S33, comparing the HSV errors of the temporary super-resolution block and the original super-resolution block at corresponding point positions, taking them as the objective function of this iterative optimization, and using a nonlinear optimization algorithm to focus on optimizing the stretching, rotation and translation parameters in the matrix of the geometric model T to minimize the objective function;

[0015] Step S34, updating the matrices of the geometric model T and the optical model M according to the nonlinear optimization result, and entering the next round of iterative optimization;

[0016] Step S35, repeating steps 32 to 34 until the objective function converges, and outputting the optimal matrix of the geometric model T and the matrix of the optical model M.

[0017] Furthermore, the step S31 includes:

[0018] Step S311, using the SIFT algorithm to extract feature points of the low-resolution image A of size Y×U to obtain R low-resolution feature points, and using the SIFT algorithm to extract feature points of the original super-resolution image J of size Q×D corresponding to the low-resolution image A to obtain F super-resolution feature points, where Y, U, R, Q, D and F are all constants, and the SIFT algorithm is a method for detecting and describing local features in an image;

[0019] Step S312, extracting local features of each low-resolution feature point to obtain a 128-dimensional SIFT descriptor a, extracting local features of each super-resolution feature point to obtain a 128-dimensional SIFT descriptor j, wherein SIFT descriptor a is a method for describing local features of low-resolution feature points, and SIFT descriptor j is a method for describing local features of super-resolution feature points;

[0020] Step S313, collect descriptors of all feature points in the low-resolution image A and the original super-resolution image J, and store them in a first list and a second list respectively, wherein the first list stores the feature point descriptor a of image A, and the second list stores the feature point descriptor j of image J, calculate the Euclidean distance between each SIFT descriptor a in the first list and each SIFT descriptor j in the second list, and establish a correspondence table from the feature points of the low-resolution image A to the feature points of the original super-resolution image J based on the descriptor j' with the smallest Euclidean distance, so as to record each feature point of the low-resolution image A and its corresponding feature point in the original super-resolution image J.

[0021] Further, the step S32 includes:

[0022] Step S321, calculating the HSV value corresponding to the feature point in the low-resolution image A;

[0023] Step S322, predicting the HSV value of the corresponding feature point of the feature point in the low-resolution image A in the original super-resolution image J according to the optical model M;

[0024] Step S323, using the HSV values ​​of the corresponding feature points corresponding to the original super-resolution image J to perform HSV value reconstruction inference calculation on all feature points in the low-resolution image A;

[0025] Step S324, constructing a temporary super-resolution reconstruction block that is consistent in size with the original super-resolution image J but has only a certain HSV value at the corresponding feature point position;

[0026] Step S325, initializing the matrix of the geometric model T;

[0027] Step S326, using the first 2×2 matrices of the matrix of the geometric model T to describe the stretching and rotation transformations, and using the third column of the matrix of the geometric model T to describe the translation transformation;

[0028] Step S327, sampling and resampling the image pixel values ​​according to the transformed coordinates to obtain an adjusted temporary super-resolution block.

[0029] Furthermore, the step S33 includes:

[0030] Step S331, obtaining the H value HL, S value SL, and V value VL of the feature point corresponding to the temporary super-resolution block, obtaining the H value HB, S value SB, and V value VB of the feature point corresponding to the original super-resolution block, calculating the difference of each channel of the HSV value of the feature point corresponding to the temporary super-resolution block and the feature point corresponding to the original super-resolution block, H difference = |HL-HB|, S difference = |SL-SB|, V difference = |VL-VB|, calculating the channel difference for all corresponding feature points and taking the square sum, objective function E = Σ(HL-HB2+SL-SB2+VL-VB2), where Σ represents the sum of all feature points, and E is the total error between the HSV values ​​of the corresponding feature points of the temporary super-resolution block and the original super-resolution block;

[0031] Step S332, using the quasi-Newton method as the nonlinear optimization algorithm, assuming that the matrix of the geometric model T contains only one translation parameter t, initialize t = 0, calculate the temporary super-resolution block and the objective function E according to the current t, calculate the first objective function E1 = E × (t + Δt) for a slight increment of t Δt, calculate the second objective function E2 = E × (t - Δt) for a slight decrement of t Δt, and calculate the gradient of t by finite difference approximation Update t using the gradient descent formula, Where α is the learning rate, t_new is the updated t, and t_new is used to repeatedly iteratively optimize t until E converges. The same process as the translation parameter optimization is repeated for the matrix rotation angle parameter θ of the geometric model T until E converges. The same process as the translation parameter and rotation angle parameter optimization is repeated for the matrix stretching parameter φ of the geometric model T until E converges to minimize the objective function E.

[0032] Furthermore, the step S1 comprises:

[0033] Step S11, evenly dividing the input low-resolution image into a plurality of non-overlapping low-resolution pixel blocks of the same size;

[0034] Step S12, extracting a color histogram of each low-resolution pixel block;

[0035] Step S13, in the super-resolution sample database, a feature matching algorithm is used for the color histogram of the low-resolution pixel block to retrieve a number of candidate super-resolution blocks with the highest matching degree therewith;

[0036] Step S14, for each low-resolution pixel block and the corresponding candidate super-resolution blocks, calculating their SSIM similarity to form a similarity result, wherein the SSIM similarity is an indicator for measuring the similarity between two images;

[0037] Step S15, sorting the similarity results, selecting a candidate super-resolution block with the largest SSIM similarity as a matching sample of the low-resolution pixel block to form an original super-resolution block.

[0038] Furthermore, the step S12 includes:

[0039] Step S121, for the n pixel values ​​contained in the low-resolution pixel block, their RGB values ​​are Ri, Gi, Bi, i=1, 2, ..., n, wherein Ri represents the red component value of the i-th pixel, Gi represents the green component value of the i-th pixel, and Bi represents the blue component value of the i-th pixel. For each pixel i, the RGB value is converted to the HSV value, Maxi=max(Ri, Gi, Bi), Mini=min(Ri, Gi, Bi), Si=(Maxi-Mini) / Maxi, Vi=Max(Ri, Gi, Bi), wherein Si is the saturation of each pixel i, and Vi is the brightness value of the color of each pixel i. According to the relationship between Maxi and Mini, the Hue angle is determined. The color segment to which degree Hi belongs, if Maxi=Ri, then Hi is between 0°-60°, if Maxi=Gi, then Hi is between 120°-180°, if Maxi=Bi, then Hi is between 240°-300°, and then calculate the Hue angle Hi according to Maxi, Mini and the formula, when Maxi=Ri or Gi, Hi=60°×(Gi-Bi) / (Maxi-Mini), when Maxi=Bi, Hi=60°×(Ri-Gi) / (Maxi-Mini)+120, when Maxi=Gi, Hi=60°×(Bi-Ri) / (Maxi-Mini)+240, save the converted HSV value, Hi, Si, Vi, i=1, 2, ..., n;

[0040] Step S122, determining the number of bins used by the three channels R, G, and B, where bin is the scale of the color histogram, using 8-16 bins to evenly divide the color value interval 0-255, and defining the color range interval of bins for the three channels R, G, and B respectively, 0-15 of the R channel is bin1, 16-31 is bin2, ..., 231-255 is bin16; the G and B channels use the same bin division interval as the R channel;

[0041] Step S123, counting the number of pixels contained in each bin of the R channel to form an R channel histogram vector; counting the number of pixels contained in each bin of the G channel to form a G channel histogram vector; counting the number of pixels contained in each bin of the B channel to form a B channel histogram vector, arranging the count values ​​of each bin of the histograms of the three channels of R, G, and B in sequence into a vector, splicing the three vectors in RGB order into a color histogram feature vector including the three colors of red, green, and blue, standardizing the feature vector so that the value of each component is within the range of 0-1, and after counting all pixels, recording the final color histogram feature vector of the low-resolution pixel block, repeatedly counting the number of pixels contained in each bin of the three channels of R, G, and B for all low-resolution pixel blocks of the image, and saving the histogram feature vectors of each low-resolution pixel block as a feature description database;

[0042] Step S124, calculating the absolute value sum of each component of the color histogram feature vector of each low-resolution pixel block, and dividing each component value by the absolute value sum, so that the sum of each component of the color histogram feature vector is 1;

[0043] Step S125, saving the normalized color histogram feature vector corresponding to each low-resolution pixel block.

[0044] Furthermore, the step S13 includes:

[0045] Step S131, for each low-resolution pixel block, using the calculated color histogram feature vector as a query condition;

[0046] Step S132, using a Euclidean distance algorithm to calculate a matching distance between a color histogram feature vector of the low-resolution pixel block and a feature vector of a super-resolution block in a super-resolution database;

[0047] Step S133, sorting the matching distances from small to large, and selecting a number of blocks with the smallest distance as candidate sets;

[0048] Step S134, calculating an average value of the candidate set, setting a matching distance threshold to 0.7 times the average value, and retaining blocks in the candidate set whose matching distances are higher than the matching distance threshold to form a plurality of candidate super-resolution blocks.

[0049] Furthermore, the step S14 includes:

[0050] Step S141, for each low-resolution pixel block and each corresponding candidate super-resolution block, extract two pixel regions of size 8×8 to form a first pixel region and a second pixel region;

[0051] Step S142, calculate the three sub-indicators of brightness value L, contrast C and structure K in the two pixel areas, L uses the average value of the V channel in HSV as the brightness average, C uses the standard deviation of the V channel, K uses the first-order autocorrelation function approximation combined with the H and S channels to calculate, the brightness L is calculated using the average brightness value V1 of the first pixel area and the average brightness value V2 of the second pixel area, the contrast C is calculated using the standard deviation σ1 of the first pixel area and the standard deviation σ2 of the contrast of the second pixel area, the first-order autocorrelation function approximation of the H and S channels is calculated for the two areas respectively, and the first-order autocorrelation function of the H channel of the first pixel area is approximated. The autocorrelation function is approximately corrH1=f(H1), the first-order autocorrelation function of the H channel of the second pixel area is approximately corrH2=f(H2), the first-order autocorrelation function of the S channel of the first pixel area is approximately corrS1=f(S1), and the first-order autocorrelation function of the S channel of the second pixel area is approximately corrS2=f(S2). The autocorrelation function results of the H and S channels are weighted averaged, S=0.5×corrH1×corrH2+0.5×corrS1×corrS2, and the relative brightness value=(2V1×V2+Q1) / (V1 2 +V2 2 +Q1), relative contrast value = (2×σ1×σ2+Q2) / (σ1 2 +σ2 2 +Q2), structural relative value = (0.5×corrH1×corrH2+0.5×corrS1×corrS2+Q3) / ((0.5×corrH1×corrH2) 2 +(0.5×corrS1×corrS2+Q3) 2 ), where Q1, Q2 and Q3 are constants, SSIM value = relative brightness value × relative contrast value × relative structure value;

[0052] Step S143, repeatedly calculating the SSIM values ​​of all candidate super-resolution blocks.

[0053] Furthermore, the step S2 comprises:

[0054] Step S21, establishing the coordinate origins of the low-resolution pixel block and the original super-resolution block, recording a translation parameter (tx, ty) representing the offset between the coordinate origin of the low-resolution pixel block and the origin of the original super-resolution block, recording a scaling parameter (sx, sy) representing the scaling ratio of the axis coordinates of the low-resolution pixel block relative to the original super-resolution block, and recording a rotation parameter θ representing the rotation angle of the coordinate axis of the low-resolution pixel block relative to the coordinate axis of the original super-resolution block. The translation parameter (tx, ty), scaling parameter (sx, sy) and rotation parameter θ are used to describe the position corresponding to an arbitrary pixel point (x, y) in the coordinates of the low-resolution pixel block in the coordinates of the original super-resolution block. The new coordinates (x', y') = (sx×x×cosθ-sy×y×sinθ+tx, sx×x×sinθ+sy×y×cosθ+ty);

[0055] Step S22, assuming that the HSV color values ​​of the low-resolution pixel block and the original super-resolution block conform to a linear transformation relationship, record a brightness scaling parameter f representing the scaling ratio of the pixel brightness of the low-resolution pixel block relative to the original super-resolution block, V'=f×V, record a matrix M' of the optical model M, describing the linear transformation relationship between the H and S values ​​of the low-resolution pixel block and the original super-resolution block in the color space [H'S'], [H'S']=M×[HS], record a saturation scaling parameter t representing the scaling ratio of the S value of the low-resolution pixel block relative to the S' value of the original super-resolution block, S'=t×S, record a hue offset h, describing the offset of the H value of the low-resolution pixel block relative to the H' value of the original super-resolution block, H'=H+h, and combine the parameters s, M', t, h to form an optical model F(I'), F(I')={s, M', t, h};

[0056] Step S23, defining the geometric parameters of the matrix of the geometric model T as a change matrix, initializing the matrix of the geometric model T as the identity matrix T', T' = [[1, 0, 0][0, 1, 0][0, 0, 1]], initializing the matrix M' of the linear mapping optical model M as a 2×2 empty matrix, M' = [[0, 0][0, 0]], and assigning an initial value of 0 to each element of M' in turn;

[0057] Step S24, from the identity matrix T', select the rotation, stretching and translation parts of the identity matrix T' as parameters, and from the matrix M' of the optical model M, select all elements of the matrix M' as parameters as learning targets;

[0058] Step S25, saving the geometric model and the optical model corresponding to each low-resolution pixel block.

[0059] Compared with the prior art, the beneficial effect of the present invention lies in that this super-resolution reconstruction method establishes a correspondence between low-resolution blocks and super-resolution blocks, utilizes nonlinear optimization to iteratively tune the geometric optical model, realizes the reconstruction of low-resolution input images into high-quality super-resolution output, and realizes real-time reconstruction of super-resolution images.

[0060] Furthermore, while maintaining the correspondence between feature points, the model parameters are continuously optimized to achieve the best match of the low super-resolution image blocks in color and spatial structure, thereby reconstructing a high-quality super-resolution image.

[0061] Furthermore, by extracting and matching image feature points, a foundation is laid for subsequently using feature point information to establish a correspondence between low-resolution images, which is beneficial to image reconstruction.

[0062] Furthermore, by establishing the color space correspondence of low super-resolution feature points and performing geometric transformation, a temporary super-resolution block is generated, laying the foundation for subsequent optimization iterations.

[0063] Furthermore, the purpose of reconstructing super-resolution images is achieved by effectively optimizing the model using a nonlinear optimization algorithm.

[0064] Furthermore, through the block matching method, the super-resolution sample reference block with the maximum match is found for each low-resolution pixel block, and a detailed correspondence between low-resolution and super-resolution blocks is established to prepare for subsequent model learning and reconstruction.

[0065] Furthermore, the color histogram method is used to obtain the compact color statistical features of each low-resolution pixel block. This feature contains the rough distribution of various color levels in the pixel block, which provides effective information for finding the most similar super-resolution block in the sample library. This low-dimensional compact feature takes into account both efficiency and effective distinction between pixel blocks.

[0066] Furthermore, through color histogram matching, a small set of super-resolution candidate blocks that are most similar to each low-resolution pixel block are quickly found in the sample library, which lays the foundation for subsequent fine matching and model learning, and effectively achieves a close connection between low-resolution input and high-quality reference output.

[0067] Furthermore, a quantitative method is used to describe and compare the similarity of image blocks with different resolutions in terms of tone and contrast, so as to find the most similar super-resolution block, thereby achieving a more accurate image super-resolution effect.

[0068] Furthermore, the established geometric model and optical model can well describe the correspondence between image blocks of different resolutions in terms of spatial position and color attributes. Learning these two types of model parameters can achieve the purpose of image decoding and reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A flowchart of the imaging method for super-resolution reconstruction according to the present invention;

[0070] Figure 2 This is a flowchart of step S1 of the imaging method for super-resolution reconstruction according to the present invention;

[0071] Figure 3 This is a flowchart of step S12 of the imaging method for super-resolution reconstruction according to the present invention;

[0072] Figure 4 This is a flowchart of step S13 of the imaging method for super-resolution reconstruction of the present invention. DETAILED DESCRIPTION

[0073] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0074] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0075] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0076] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0077] See also Figure 1-4 As shown, Figure 1 It is a flowchart of the imaging method of super-resolution reconstruction of the present invention in an embodiment; Figure 2 It is a flowchart of step S1 of the imaging method of super-resolution reconstruction of the present invention in an embodiment; Figure 3 It is a flowchart of step S12 of the imaging method for super-resolution reconstruction of the present invention in an embodiment; Figure 4 4 is a flowchart of step S13 of the imaging method for super-resolution reconstruction according to the present invention in an embodiment.

[0078] like Figure 1 As shown, a super-resolution reconstruction imaging method comprises the following steps:

[0079] Step S1, dividing the low-resolution image into a plurality of low-resolution pixel blocks, and searching a pre-stored super-resolution block having the highest similarity to each of the low-resolution pixel blocks in a pre-established super-resolution database to form an original super-resolution block;

[0080] Step S2, establishing a geometric model and an optical model for each low-resolution pixel block;

[0081] Step S3, iteratively optimizing stretching parameters, rotation parameters, and translation parameters corresponding to the geometric model, reconstructing the low-resolution pixel block by using the optimized geometric model and the optical model to obtain a plurality of temporary super-resolution reconstruction blocks, reconstructing the plurality of temporary super-resolution reconstruction blocks by using the optical model to obtain a plurality of temporary super-resolution blocks, and determining the matching degree between the plurality of temporary super-resolution blocks and the original super-resolution blocks;

[0082] Step S4, assembling each of the temporary super-resolution blocks with the highest matching degree to obtain a complete super-resolution image.

[0083] The low-resolution image is divided into multiple pixel blocks, and the super-resolution block that is most similar to each pixel block is found. For each low-resolution pixel block, model parameters describing its geometric and optical transformations are established. The model parameters are iteratively adjusted using a nonlinear optimization algorithm to maximize the matching degree between the reconstructed temporary super-resolution block and the block. The optimized super-resolution blocks that match each low-resolution pixel block are spliced ​​to output the final super-resolution result.

[0084] This super-resolution reconstruction method establishes a correspondence between low-resolution blocks and super-resolution blocks, uses nonlinear optimization to iteratively tune the geometric optical model, and reconstructs low-resolution input images into high-quality super-resolution outputs, thereby achieving real-time reconstruction of super-resolution images.

[0085] like Figure 2 As shown, further, the step S1 includes:

[0086] Step S11, evenly dividing the input low-resolution image into a plurality of non-overlapping low-resolution pixel blocks of the same size;

[0087] Step S12, extracting a color histogram of each low-resolution pixel block;

[0088] Step S13, in the super-resolution sample database, a feature matching algorithm is used for the color histogram of the low-resolution pixel block to retrieve a number of candidate super-resolution blocks with the highest matching degree therewith;

[0089] Step S14, calculating the SSIM similarity between each low-resolution pixel block and the corresponding candidate super-resolution blocks to form a similarity result;

[0090] Step S15, sorting the similarity results, selecting a candidate super-resolution block with the largest SSIM similarity as a matching sample of the low-resolution pixel block to form an original super-resolution block.

[0091] The low-resolution image is evenly divided into multiple pixel blocks, and the color histogram features are extracted for each pixel block. Then, candidate super-resolution blocks are retrieved in the sample library based on feature matching, and the SSIM similarity between each candidate block and the low-resolution pixel block is calculated. By selecting the candidate block with the greatest similarity, the original super-resolution block that best matches it is determined for each low-resolution pixel block.

[0092] Through the block matching method, the super-resolution sample reference block with the maximum match is found for each low-resolution pixel block, and a detailed correspondence between low-resolution and super-resolution blocks is established to prepare for subsequent model learning and reconstruction.

[0093] like Figure 3 As shown, further, the step S12 includes:

[0094] Step S121, performing color space conversion from RGB to HSV on each low-resolution pixel block;

[0095] For the n pixel values ​​contained in the low-resolution pixel block, their RGB values ​​are Ri, Gi, Bi, i=1, 2, ..., n, wherein: Ri represents the red component value of the i-th pixel, Gi represents the green component value of the i-th pixel, and Bi represents the blue component value of the i-th pixel;

[0096] Convert RGB to HSV for each pixel i, Maxi = max(Ri, Gi, Bi), Mini = min(Ri, Gi, Bi), Si = (Maxi-Mini) / Maxi, Vi = Max(Ri, Gi, Bi), where Si is the saturation of each pixel i, Vi is the brightness value of the color of each pixel i, and according to the relationship between Maxi and Mini, determine the color segment to which the Hue angle Hi belongs;

[0097] If Maxi = Ri, then Hi is between 0° and 60°;

[0098] If Maxi=Gi, then Hi is between 120° and 180°;

[0099] If Maxi=Bi, then Hi is between 240° and 300°;

[0100] Then calculate the Hue angle Hi based on Maxi, Mini and the formula;

[0101] When Maxi=Ri or Gi, Hi=60°×(Gi-Bi) / (Maxi-Mini);

[0102] When Maxi=Bi, Hi=60°×(Ri-Gi) / (Maxi-Mini)+120;

[0103] When Maxi=Gi, Hi=60°×(Bi-Ri) / (Maxi-Mini)+240;

[0104] Save the converted HSV values, Hi, Si, Vi, i=1, 2, ..., n;

[0105] Step S122, determining the number of bins used by the three channels of R, G, and B, and using 8-16 bins to evenly divide the color value range 0-255;

[0106] Define the color range of bins for the three channels R, G, and B respectively. For the R channel, 0-15 is bin1, 16-31 is bin2, ..., 231-255 is bin16; the G and B channels use the same bin division interval as the R channel;

[0107] Step S123, counting the number of pixels contained in each bin of the R channel to form an R channel histogram vector; counting the number of pixels contained in each bin of the G channel to form a G channel histogram vector; counting the number of pixels contained in each bin of the B channel to form a B channel histogram vector;

[0108] Arrange the bin count values ​​of the histograms of the three channels R, G, and B into a vector, and concatenate the three vectors in RGB order into a color histogram feature vector including red, green, and blue.

[0109] Standardize the eigenvector so that the value of each component is in the range of 0-1;

[0110] After counting all pixels, the final color histogram feature vector of the low-resolution pixel block is recorded;

[0111] Repeat the counting of the number of pixels contained in each bin of the three channels R, G, and B for all low-resolution pixel blocks of the image;

[0112] Save the histogram feature vector of each low-resolution pixel block as a feature description database;

[0113] Step S124, calculating the absolute value sum of each component of the color histogram feature vector of each low-resolution pixel block;

[0114] Dividing each component value by the absolute value sum so that the sum of each component of the color histogram feature vector is 1;

[0115] Step S125, saving the normalized color histogram feature vector corresponding to each low-resolution pixel block.

[0116] This step first performs RGB to HSV color space conversion on each low-resolution pixel block, then uses the color histogram method to extract the color distribution characteristics of each pixel block, evenly divides each pixel block into multiple color connected domain bins, counts the number of pixels contained in each bin, forms the histogram vectors of the three channels of R, G, and B, and finally splices them into a standardized color histogram feature vector. In this way, a compact statistical description of the color distribution of each low-resolution pixel block is extracted.

[0117] The color histogram method is used to obtain the compact color statistical features of each low-resolution pixel block. This feature contains the rough distribution of various color levels in the pixel block, which provides effective information for finding the most similar super-resolution block in the sample library. This low-dimensional compact feature takes into account both efficiency and effective distinction between pixel blocks.

[0118] like Figure 4 As shown, further, the step S13 includes:

[0119] Step S131, for each low-resolution pixel block, using the calculated color histogram feature vector as a query condition;

[0120] Step S132, using a Euclidean distance algorithm to calculate a matching distance between a color histogram feature vector of the low-resolution pixel block and a feature vector of a super-resolution block in a super-resolution database;

[0121] Step S133, sorting the matching distances from small to large, and selecting a number of blocks with the smallest distance as candidate sets;

[0122] Step S134, calculating an average value of the candidate set, setting a matching distance threshold to 0.7 times the average value, and retaining blocks in the candidate set whose matching distances are higher than the matching distance threshold to form a plurality of candidate super-resolution blocks.

[0123] The Euclidean distance algorithm is used to calculate the matching distance between each low-resolution pixel block and all blocks, and several blocks with the smallest distance are selected as the preliminary candidate set. The threshold limit is further set to remove blocks with a long distance to obtain a more suitable super-resolution candidate block set.

[0124] Through color histogram matching, a small set of super-resolution candidate blocks that are most similar to each low-resolution pixel block are quickly found in the sample library. This lays the foundation for subsequent fine matching and model learning, and effectively achieves a close connection between low-resolution input and high-quality reference output.

[0125] Furthermore, the step S14 includes:

[0126] Step S141, extracting two pixel regions of size 8x8 from each low-resolution pixel block and each corresponding candidate super-resolution block to form a first pixel region and a second pixel region;

[0127] Step S142, calculating the three sub-indicators of brightness value L, contrast C and structure K in the two pixel regions, where L uses the average value of the V channel in HSV as the brightness average, C uses the standard deviation of the V channel, and K uses the first-order autocorrelation function approximation combined with the H and S channels for calculation;

[0128] Calculate the brightness L using the average brightness value V1 of the first pixel area and the average brightness value V2 of the second pixel area;

[0129] Calculate the contrast C using the standard deviation σ1 of the first pixel area and the standard deviation σ2 of the contrast of the second pixel area;

[0130] The first-order autocorrelation function of the H and S channels of the two regions is calculated respectively. The first-order autocorrelation function of the H channel of the first pixel region is approximately corrH1=f(H1), the first-order autocorrelation function of the H channel of the second pixel region is approximately corrH2=f(H2), the first-order autocorrelation function of the S channel of the first pixel region is approximately corrS1=f(S1), and the first-order autocorrelation function of the S channel of the second pixel region is approximately corrS2=f(S2);

[0131] Perform weighted averaging on the autocorrelation function results of the H and S channels;

[0132] S=0.5×corrH1×corrH2+0.5×corrS1×corrS2;

[0133] Relative brightness value = (2V1×V2+Q1) / (V1 2 +V2 2 +Q1);

[0134] Relative contrast value = (2×σ1×σ2+Q2) / (σ1 2 +σ2 2 +Q2);

[0135] Structural relative value = (0.5×corrH1×corrH2+0.5×corrS1×corrS2+Q3) / ((0.5×corrH1×corrH2) 2 +(0.5×corrS1×corrS2+Q3) 2 ), where Q1, Q2 and Q3 are constants;

[0136] SSIM value = relative brightness value × relative contrast value × relative structure value;

[0137] Step S143, repeatedly calculating the SSIM values ​​of all candidate super-resolution blocks.

[0138] Two 8×8 pixel regions are extracted from the low-resolution image block and the candidate super-resolution block, and the values ​​of the three sub-indicators of brightness, contrast and structure of the two regions are calculated. The relative values ​​of brightness, contrast and structure are calculated according to the sub-indicator value formula to obtain the SSIM similarity value.

[0139] The similarity of the tone and contrast of image blocks with different resolutions is described and compared in a quantitative way to find the most similar super-resolution block, thereby achieving a more accurate image super-resolution effect.

[0140] Furthermore, the step S2 comprises:

[0141] Step S21, establishing the coordinate origin of the low-resolution pixel block and the original super-resolution block;

[0142] Record the translation parameter (tx, ty) representing the offset between the low-resolution pixel block coordinate origin and the original super-resolution block origin;

[0143] Record the scaling parameters (sx, sy) representing the scaling ratio of the axis coordinates of the low-resolution pixel block relative to the original super-resolution block;

[0144] Recording a rotation parameter θ representing the rotation angle of the low-resolution pixel block coordinate axis relative to the original super-resolution block coordinate axis;

[0145] Using the translation parameter (tx, ty), the scaling parameter (sx, sy) and the rotation parameter θ, the corresponding position of any pixel point (x, y) in the low-resolution pixel block coordinates in the original super-resolution block coordinates is described;

[0146] New coordinates (x', y') = (sx×x×cosθ-sy×y×sinθ+tx, sx×x×sinθ+sy×y×cosθ+ty);

[0147] Step S22, assuming that the HSV color values ​​of the low-resolution pixel block and the original super-resolution block conform to a linear transformation relationship, record a brightness scaling parameter f representing the scaling ratio of the pixel brightness of the low-resolution pixel block relative to the original super-resolution block, V'=f×V, record a matrix M' of the optical model M, describing the linear transformation relationship between the H and S values ​​of the low-resolution pixel block and the original super-resolution block in the color space [H'S'], [H'S']=M×[HS], record a saturation scaling parameter t representing the scaling ratio of the S value of the low-resolution pixel block relative to the S' of the original super-resolution block, S'=t×S, record a hue offset h, describing the offset of the H value of the low-resolution pixel block relative to the H' value of the original super-resolution block, H'=H+h, and combine the parameters s, M', t, h to form an optical model F(I'), F(I')={s, M', t, h};

[0148] Step S23, defining the geometric parameters of the matrix of the geometric model T as a change matrix, initializing the matrix of the geometric model T as the identity matrix T', T' = [[1, 0, 0][0, 1, 0][0, 0, 1]], initializing the matrix M' of the linear mapping optical model M as a 2×2 empty matrix, M' = [[0, 0][0, 0]], and assigning an initial value of 0 to each element of M' in turn;

[0149] Step S24, from the identity matrix T', select the rotation, stretching and translation parts of the identity matrix T' as parameters, and from the matrix M' of the optical model M, select all elements of the matrix M' as parameters as learning targets;

[0150] Step S25, saving the geometric model and the optical model corresponding to each low-resolution pixel block.

[0151] Establish the correspondence between the low-resolution image block and the original super-resolution block in the coordinate system, record the translation, scaling, and rotation geometric transformation parameters, assume that the low-resolution and super-resolution image blocks conform to linear transformation in the HSV color space, record the optical model parameters such as brightness scaling and color space mapping, define the geometric transformation parameters as the transformation matrix T', and the optical model parameters as the matrix M', as the target parameters for subsequent learning.

[0152] The established geometric model and optical model can well describe the correspondence between the spatial position and color attributes of image blocks of different resolutions. Learning these two types of model parameters can achieve the purpose of image decoding and reconstruction.

[0153] Furthermore, the step S3 comprises:

[0154] Step S31, extracting feature points in the low-resolution pixel block and establishing a matching relationship with corresponding feature points in the original super-resolution image;

[0155] Step S32, reconstructing the HSV value of the low-resolution pixel block using the optical model M according to the feature point matching relationship to obtain a temporary super-resolution reconstruction block, and performing stretching, rotation and translation transformation on the temporary super-resolution reconstruction block according to the geometric model T to obtain an adjusted temporary super-resolution block;

[0156] Step S33, comparing the HSV errors of the temporary super-resolution block and the original super-resolution block at corresponding point positions, taking them as the objective function of this iterative optimization, and using a nonlinear optimization algorithm to focus on optimizing the stretching, rotation and translation parameters in the matrix of the geometric model T to minimize the objective function;

[0157] Step S34, updating the matrices of the geometric model T and the optical model M according to the nonlinear optimization result, and entering the next round of iterative optimization;

[0158] Step S35, repeating steps 32 to 34 until the objective function converges, and outputting the optimal matrix of the geometric model T and the matrix of the optical model M.

[0159] Extract feature points from low-resolution image blocks, establish a matching relationship with the original super-resolution image, and reconstruct a temporary super-resolution image block based on feature point matching using the optical model M, and convert it according to the geometric model T. Take the difference between the temporary block and the original block at the corresponding point position as the error function, use the optimization algorithm to optimize the matrix parameters of the geometric model T to minimize the error, repeat the optimization process, update the matrices of the geometric model T and the optical model M, until the error converges, and output the optimal model parameters.

[0160] While maintaining the correspondence between feature points, the model parameters are continuously optimized to achieve the best match of low super-resolution image blocks in color and spatial structure, thereby reconstructing high-quality super-resolution images.

[0161] Furthermore, the step S31 includes:

[0162] Step S311, using the SIFT algorithm to extract feature points of the low-resolution image A of size Y×U to obtain R low-resolution feature points, and using the SIFT algorithm to extract feature points of the original super-resolution image J of size Q×D corresponding to the low-resolution image A to obtain F super-resolution feature points, where Y, U, R, Q, D and F are all constants, and the SIFT algorithm is a method for detecting and describing local features in an image;

[0163] Step S312, step S312, extracting local features of each low-resolution feature point to obtain a 128-dimensional SIFT descriptor a, extracting local features of each super-resolution feature point to obtain a 128-dimensional SIFT descriptor j, wherein SIFT descriptor a is a method for describing local features of low-resolution feature points, and SIFT descriptor j is a method for describing local features of super-resolution feature points;

[0164] Step S313, collecting descriptors of all feature points in the low-resolution image A and the original super-resolution image J, and storing them in the first list and the second list respectively;

[0165] The first list stores the feature point descriptor a of the image A;

[0166] The second list stores the feature point descriptor j of image J;

[0167] For each SIFT descriptor a in the first list, the Euclidean distance between it and each SIFT descriptor j in the second list is calculated, and a correspondence table between the feature points of the low-resolution image A and the feature points of the original super-resolution image J is established according to the descriptor j' with the smallest Euclidean distance, so as to record each feature point of the low-resolution image A and its corresponding feature point in the original super-resolution image J.

[0168] The SIFT algorithm is used to extract feature points from the low-resolution image and the original super-resolution image, and the descriptors of the feature points are calculated. All feature point descriptors of the low-resolution and super-resolution images are stored in two lists respectively. For each feature point descriptor of the low-resolution image, the Euclidean distance with each descriptor of the super-resolution image is calculated, the corresponding point with the smallest distance is found, and a matching relationship table of low-resolution and super-resolution feature points is established.

[0169] By extracting and matching image feature points, the foundation is laid for the subsequent use of feature point information to establish the correspondence between low-resolution super-resolution images, which is conducive to image reconstruction.

[0170] Further, the step S32 includes:

[0171] Step S321, calculating the HSV value corresponding to the feature point in the low-resolution image A;

[0172] Step S322, predicting the HSV value of the corresponding feature point of the feature point in the low-resolution image A in the original super-resolution image J according to the optical model M;

[0173] Step S323, using the HSV values ​​of the corresponding feature points corresponding to the original super-resolution image J to perform HSV reconstruction inference calculation on all feature points in the low-resolution image A;

[0174] Step S324, constructing a temporary super-resolution reconstruction block that is consistent in size with the original super-resolution image J but has only a certain HSV value at the corresponding feature point position;

[0175] Step S325, initializing the matrix of the geometric model T;

[0176] Step S326, using the first 2×2 matrices of the matrix of the geometric model T to describe the stretching and rotation transformations, and using the third column of the matrix of the geometric model T to describe the translation transformation;

[0177] Step S327, sampling and resampling the image pixel values ​​according to the transformed coordinates to obtain an adjusted temporary super-resolution block.

[0178] According to the HSV value of the low-resolution feature point and the optical model M, predict its corresponding HSV value in the super-resolution image. Use the predicted value to reconstruct the HSV in the low-resolution image and construct a temporary super-resolution block that is the same size as the super-resolution image but has values ​​only at the feature point position. Initialize the matrix of the geometric model T, use the geometric model T to describe the stretching, rotation and translation transformations, and perform coordinate transformation on the temporary super-resolution block.

[0179] By establishing the color space correspondence of low super-resolution feature points and performing geometric transformation, a temporary super-resolution block is generated, laying the foundation for subsequent optimization iterations.

[0180] Furthermore, the step S33 includes:

[0181] Step S331, obtaining the H value HL, the S value SL, and the V value VL of the feature point corresponding to the temporary super-resolution block;

[0182] Obtain the H value HB, the S value SB, and the V value VB of the feature point corresponding to the original super-resolution block;

[0183] Calculate the difference of each channel of the HSV value of the feature point corresponding to the temporary super-resolution block and the feature point corresponding to the original super-resolution block, H difference = |HL-HB|, S difference = |SL-SB|, V difference = |VL-VB|, calculate the channel difference of all corresponding feature points and take the square sum, objective function E = Σ(HL-HB 2 +SL-SB 2 +VL-VB 2 ), where Σ represents the sum of all feature points, and E is the total error between the HSV values ​​of the corresponding feature points of the temporary super-resolution block and the original super-resolution block;

[0184] Step S332, using the quasi-Newton method as a nonlinear optimization algorithm, assuming that the matrix of the geometric model T contains only one translation parameter t, initializing t=0, and calculating a temporary super-resolution block and an objective function E according to the current t;

[0185] For the micro-increment Δt, calculate the first objective function E1=E×(t+Δt);

[0186] For a slight decrease in t by Δt, calculate the second objective function E2=E×(t-Δt);

[0187] The gradient of t is approximated by finite differences

[0188] Update t using the gradient descent formula;

[0189] Where α is the learning rate, and t_new is the updated t;

[0190] Use t_new to repeatedly iterate and optimize t until E converges;

[0191] Repeat the same process as the translation parameter optimization for the matrix rotation angle parameter θ of the geometric model T until E converges;

[0192] The same process as the optimization of the translation parameters and rotation angle parameters is repeated for the matrix stretching parameter φ of the geometric model T until E converges to minimize the objective function E.

[0193] The sum of the differences in the HSV values ​​of the temporary reconstructed block and the original block at the corresponding feature point positions is calculated as the objective function E. The quasi-Newton optimization method is used to cyclically optimize the parameters in the matrix of the geometric model T, including translation, rotation, and stretching, and the parameters are updated with small changes each time. The value of the objective function E is calculated based on the current parameters, and the parameter gradient is obtained by differentiation. The gradient descent formula is used to iteratively update the parameters so that the objective function E gradually converges to the minimum. By continuously optimizing the parameters of the geometric model T, the temporary reconstructed block can better match the original block in terms of tone and structure, thereby improving the image quality.

[0194] The purpose of reconstructing super-resolution images is achieved by effectively optimizing the model using a nonlinear optimization algorithm.

[0195] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0196] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A super-resolution reconstruction imaging method, characterized in that: include: Step S1, dividing the low-resolution image into a plurality of low-resolution pixel blocks, and searching a pre-stored super-resolution block having the highest similarity to each of the low-resolution pixel blocks in a pre-established super-resolution database to form an original super-resolution block; Step S2, establishing a geometric model and an optical model for each low-resolution pixel block; Step S3, iteratively optimizing stretching parameters, rotation parameters, and translation parameters corresponding to the geometric model, reconstructing the low-resolution pixel block by using the optimized geometric model and the optical model to obtain a plurality of temporary super-resolution reconstruction blocks, reconstructing the plurality of temporary super-resolution reconstruction blocks by using the optical model to obtain a plurality of temporary super-resolution blocks, and determining the matching degree between the plurality of temporary super-resolution blocks and the original super-resolution blocks; Step S4, assembling each of the temporary super-resolution blocks with the highest matching degree to obtain a complete super-resolution image; The step S1 comprises: Step S11, evenly dividing the input low-resolution image into a plurality of non-overlapping low-resolution pixel blocks of the same size; Step S12, extracting a color histogram of each low-resolution pixel block; Step S13, in the super-resolution sample database, a feature matching algorithm is used for the color histogram of the low-resolution pixel block to retrieve a number of candidate super-resolution blocks with the highest matching degree therewith; Step S14, for each low-resolution pixel block and the corresponding candidate super-resolution blocks, calculating their SSIM similarity to form a similarity result, wherein the SSIM similarity is an indicator for measuring the similarity between two images; Step S15, sorting the similarity results, selecting a candidate super-resolution block with the largest SSIM similarity as a matching sample of the low-resolution pixel block to form an original super-resolution block.

Images