Image dictionary generation method, image reconstruction method and related device

Abstract

The present application discloses a method for generating an image dictionary, an image reconstruction method and a related device, comprising: obtaining multiple initial image groups; wherein each initial image group includes a first image with a resolution exceeding a resolution threshold, and a second image below the resolution threshold generated based on the first image; performing non-subsampled contourlet transform on the first image and the second image in each initial image group according to different transformation parameters, respectively, to obtain multiple first sub-images corresponding to the first image and multiple second sub-images corresponding to the second image; constructing multiple training image groups based on the multiple first sub-images and the multiple second sub-images; wherein the same training image group includes the first sub-image and the second sub-image with the same transformation parameters in the same initial image group; based on each training image group, generating a first dictionary and a second dictionary corresponding to each transformation parameter. The above scheme can improve the reconstruction effect of image reconstruction based on the first dictionary and the second dictionary.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image dictionary generation method, an image reconstruction method and related devices. Background Art

[0002] With the continuous development of the field of image processing, the reconstruction of low-resolution images has become an important research direction. Among them, learning-based methods have received widespread attention. Existing learning-based methods for reconstructing images are mainly based on dictionaries to reconstruct images. However, in the stage of obtaining dictionaries, due to insufficient decomposition of image information, there are too many elements in the dictionary, and the final image reconstruction has high computational complexity and poor detail reconstruction effect. In view of this, how to improve the reconstruction effect of image reconstruction based on the first dictionary and the second dictionary has become an urgent problem to be solved. Summary of the invention

[0003] The main technical problem solved by the present application is to provide a method for generating an image dictionary, an image reconstruction method and related devices, which can improve the reconstruction effect of image reconstruction based on a first dictionary and a second dictionary.

[0004] To solve the above technical problems, the first aspect of the present application provides a method for generating an image dictionary, the method comprising: obtaining a plurality of initial image groups; wherein each of the initial image groups comprises a first image having a resolution exceeding a resolution threshold, and a second image having a resolution lower than the resolution threshold generated based on the first image; performing a non-subsampled contourlet transform on the first image and the second image in each of the initial image groups according to different transformation parameters, respectively, to obtain a plurality of first sub-images corresponding to the first image and a plurality of second sub-images corresponding to the second image; constructing a plurality of training image groups based on the plurality of the first sub-images and the plurality of the second sub-images; wherein the same training image group comprises the first sub-image and the second sub-image having the same transformation parameters in the same initial image group; based on each of the training image groups, generating a first dictionary and a second dictionary corresponding to each of the transformation parameters respectively; wherein the first dictionary corresponds to images having a resolution higher than the resolution threshold, and the second dictionary corresponds to images having a resolution lower than the resolution threshold.

[0005] To solve the above technical problems, the second aspect of the present application provides an image reconstruction method, which includes: obtaining an image to be reconstructed with a resolution lower than a resolution threshold; performing a non-subsampled contourlet transform on the image to be reconstructed according to different transformation parameters to obtain a plurality of sub-images to be reconstructed corresponding to the image to be reconstructed; dividing the sub-image to be reconstructed corresponding to each transformation parameter into a plurality of image blocks to be reconstructed; converting the image blocks to be reconstructed into one-dimensional vectors, and determining the sparse coefficients corresponding to each one-dimensional vector in a second dictionary corresponding to each one-dimensional vector based on an orthogonal matching pursuit algorithm; generating a reconstructed image block corresponding to each image block to be reconstructed based on a first dictionary corresponding to each one-dimensional vector and the sparse coefficients corresponding to each one-dimensional vector, and splicing each reconstructed image block in a segmentation order to generate a reconstructed image corresponding to the image to be reconstructed; wherein the first dictionary and the second dictionary corresponding to the one-dimensional vector match the transformation parameters corresponding to the sub-image to be reconstructed corresponding to the one-dimensional vector, and the first dictionary and the second dictionary corresponding to each transformation parameter are obtained based on the method described in the first aspect above.

[0006] To solve the above technical problems, the third aspect of the present application provides an electronic device, which includes: a memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method described in the first aspect or the second aspect.

[0007] In order to solve the above technical problems, the fourth aspect of the present application provides a computer storage medium on which program data is stored, and when the program data is executed by a processor, the method described in the first aspect or the second aspect is implemented.

[0008] In the above scheme, the initial image group includes a first image with a resolution exceeding a resolution threshold, and a second image with a resolution below the resolution threshold generated based on the first image, and the corresponding first and second images form an initial image group. The first image and the second image in the initial image group are subjected to non-subsampled contourlet transform according to different transformation parameters to obtain a plurality of first sub-images and second sub-images, wherein the non-subsampled contourlet transform has the characteristics of translation invariance, multi-resolution and multi-directionality, and can decompose the image more fully, and a plurality of training image groups are constructed based on the plurality of first sub-images and the plurality of second sub-images, wherein the same training image group includes the first sub-image and the second sub-image with the same transformation parameters in the same initial image group, and therefore, the same training image group includes the first sub-image after the high-resolution first image is transformed and the second sub-image after the low-resolution second image is transformed, and based on each training image group, a first dictionary and a second dictionary corresponding to each transformation parameter are generated respectively, so that the first dictionary and the second dictionary can be more sparse, and the first dictionary and the second dictionary corresponding to the same transformation parameter correspond to the high-resolution image and the low-resolution image respectively, and then the low-resolution image can be reconstructed based on the first dictionary and the second dictionary, thereby improving the reconstruction effect of image reconstruction based on the first dictionary and the second dictionary. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0010] Figure 1 It is a flowchart of an implementation method of an image dictionary generation method of the present application;

[0011] Figure 2 It is a flowchart of another implementation method of the image dictionary generation method of the present application;

[0012] Figure 3 It is a flowchart of an implementation method of the image reconstruction method of the present application;

[0013] Figure 4 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0014] Figure 5 It is a structural schematic diagram of an implementation method of a computer storage medium of the present application. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0016] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0017] See also Figure 1 , Figure 1 1 is a flow chart of an embodiment of a method for generating an image dictionary of the present application, the method comprising:

[0018] S101: Obtain multiple initial image groups.

[0019] Specifically, each initial image group includes a first image whose resolution exceeds a resolution threshold, and a second image whose resolution is lower than the resolution threshold and is generated based on the first image.

[0020] In one application, a plurality of first images having resolutions exceeding a resolution threshold are obtained, the first images are downsampled so as to be reduced to a corresponding preset ratio according to the set resolution, the first images are converted into second images having a resolution lower than the resolution threshold, and the initial first images are retained, and the corresponding first images and second images form an initial image group, so that a plurality of initial image groups can be obtained based on the plurality of first images.

[0021] In one application scenario, the resolution threshold is set to 180 pixels per inch (PPI). After obtaining a first image whose resolution exceeds the resolution threshold, the resolution of the second image is reduced to one-Nth of the first image so that the resolution of the second image is lower than the classification threshold, and the first image and the corresponding second image are taken as an initial image group.

[0022] S102: Perform non-subsampled contourlet transform on the first image and the second image in each initial image group according to different transformation parameters to obtain a plurality of first sub-images corresponding to the first image and a plurality of second sub-images corresponding to the second image.

[0023] Specifically, non-subsampled contourlet transform (NSCT) is performed on the first image and the second image in each initial image group respectively, and different transform parameters are used for the first image and the second image in the same initial image group when performing NSCT transform, and the transform parameters are used for other initial image groups, and NSCT transform is performed on the first image and the second image in the initial image group to obtain a plurality of first sub-images corresponding to the first image and a plurality of second sub-images corresponding to the second image. Among them, non-subsampled contourlet transform has the characteristics of translation invariance, multi-resolution and multi-directionality, and can decompose the image more fully.

[0024] In one application, a one-layer four-directional NSCT transformation is performed on the first image and the second image in each initial image group to obtain at least one high-frequency image and at least one low-frequency image corresponding to the first image, and at least one high-frequency image and at least one low-frequency image corresponding to the second image, wherein the low-frequency image is obtained based on a low-pass sub-band and the high-frequency image is obtained based on a band-pass sub-band, thereby converting the first image into multiple first sub-images and converting the second image into multiple second sub-images to obtain sub-images with different features in the frequency domain, thereby extracting more refined features.

[0025] S103: constructing a plurality of training image groups based on the plurality of first sub-images and the plurality of second sub-images; wherein the same training image group includes the first sub-images and the second sub-images having the same transformation parameters in the same initial image group.

[0026] Specifically, the first sub-image and the second sub-image with the same transformation parameter in the same initial image group are grouped into the same training image group, thereby constructing a plurality of training image groups.

[0027] In an application scenario, first sub-images and second sub-images with the same transformation parameters in each initial image group are matched to obtain each training image group.

[0028] S104: Based on each training image group, generate a first dictionary and a second dictionary corresponding to each transformation parameter.

[0029] Specifically, the first sub-image in the training image group is divided into a plurality of image blocks, and the second sub-image is divided into a plurality of image blocks, and the number of image blocks corresponding to the first sub-image is the same as the number of image blocks corresponding to the second sub-image, to obtain an image block pair consisting of an image block with a resolution higher than a resolution threshold and an image block with a resolution lower than the resolution threshold, wherein the transformation parameters corresponding to each image block pair are the same.

[0030] Furthermore, based on the compressed sensing theory, the high-resolution image blocks and the low-resolution image blocks in the image block pair are converted into one-dimensional vectors, and are spliced ​​into an initial vector in sequence, and the dimension of the initial vector is increased, and the increased dimension is filled with zero values ​​to obtain a joint dictionary to make the atoms in the joint dictionary more sparse, and the joint dictionary is corrected, and the sparse coefficients corresponding to the joint dictionary and the atoms in the joint dictionary are updated to obtain an optimized joint dictionary, and the joint dictionary is divided into a first dictionary and a second dictionary in the splicing order, wherein the first dictionary corresponds to images with a resolution higher than a resolution threshold, and the second dictionary corresponds to images with a resolution lower than the resolution threshold.

[0031] In an application scenario, the first dictionary and the second dictionary corresponding to the same transformation parameter correspond to a high-resolution image and a low-resolution image respectively, and the first dictionary and the second dictionary have the same corresponding sparse coefficients in the generation stage. When the low-resolution image is reconstructed using the first dictionary and the second dictionary, after the sparse coefficients of the image with a resolution lower than the resolution threshold are determined from the second dictionary, the low-resolution image can be converted into an image with a resolution higher than the resolution threshold based on the sparse coefficients determined from the second dictionary and the first dictionary corresponding to the same transformation parameter, thereby performing an inverse transformation based on the NSCT transformation to reconstruct the low-resolution image into a high-resolution image.

[0032] In the above scheme, the initial image group includes a first image with a resolution exceeding a resolution threshold, and a second image with a resolution below the resolution threshold generated based on the first image, and the corresponding first and second images form an initial image group. The first image and the second image in the initial image group are subjected to non-subsampled contourlet transform according to different transformation parameters to obtain a plurality of first sub-images and second sub-images, wherein the non-subsampled contourlet transform has the characteristics of translation invariance, multi-resolution and multi-directionality, and can decompose the image more fully, and a plurality of training image groups are constructed based on the plurality of first sub-images and the plurality of second sub-images, wherein the same training image group includes the first sub-image and the second sub-image with the same transformation parameters in the same initial image group, and therefore, the same training image group includes the first sub-image after the high-resolution first image is transformed and the second sub-image after the low-resolution second image is transformed, and based on each training image group, a first dictionary and a second dictionary corresponding to each transformation parameter are generated respectively, so that the first dictionary and the second dictionary can be more sparse, and the first dictionary and the second dictionary corresponding to the same transformation parameter correspond to the high-resolution image and the low-resolution image respectively, and then the low-resolution image can be reconstructed based on the first dictionary and the second dictionary, thereby improving the reconstruction effect of image reconstruction based on the first dictionary and the second dictionary.

[0033] See also Figure 2 , Figure 2FIG. 1 is a flow chart of another embodiment of a method for generating an image dictionary of the present application, the method comprising:

[0034] S201: Obtain multiple first images whose resolution exceeds a resolution threshold, and perform noise processing on each of the first images.

[0035] Specifically, a plurality of first images with resolutions exceeding a resolution threshold are obtained, and Gaussian noise and impulse noise are added to each of the first images so that the first images obtain more interference data.

[0036] S202: Down-sampling each first image after noise processing to obtain a second image corresponding to each first image and having a resolution lower than a resolution threshold, so as to generate a plurality of initial image groups, and the ratio between the pixel sizes of the first image and the second image is a preset ratio.

[0037] Specifically, each first image after noise addition is downsampled to reduce the resolution of the first image, and second images with resolutions lower than a resolution threshold corresponding to each first image are obtained. The corresponding first images and second images are grouped into an initial image group to obtain multiple initial image groups, and the ratio of pixel sizes between the first image and the second image in the initial image group is a preset ratio.

[0038] In one application scenario, the first image after noise addition is downsampled at a preset ratio to generate a second image corresponding to the first image, and the first image and the corresponding second image are combined into an initial image group, so that the first image and the second image in each initial image group originate from the same image, and the pixel size between the first image and the second image corresponds to a preset ratio.

[0039] S203: Perform non-subsampled contourlet transform on the first image and the second image in each initial image group according to different transformation parameters to obtain a plurality of first sub-images corresponding to the first image and a plurality of second sub-images corresponding to the second image.

[0040] Specifically, in each initial image group, NSCT transformation is performed on the first image and the second image according to different transformation parameters to obtain multiple first sub-images corresponding to the first image and multiple second sub-images corresponding to the second image, wherein the transformation parameters include the number of transformation layers and the transformation direction.

[0041] In one application, a non-subsampled contourlet transform is performed on the first image in each initial image group according to different numbers of transform layers and transform directions to obtain at least one first low-frequency sub-image and at least one first high-frequency sub-image corresponding to the first image as multiple first sub-images; and a non-subsampled contourlet transform is performed on the second image in each initial image group according to different numbers of transform layers and transform directions to obtain at least one second low-frequency sub-image and at least one second high-frequency sub-image corresponding to the second image as multiple second sub-images.

[0042] Specifically, non-subsampled contourlet transform is performed on the first image and the second image according to different transform layers and transform directions, and frequency domain texture decomposition is performed on the horizontal, vertical and diagonal directions of the image, so as to obtain at least one first low-frequency sub-image and at least one first high-frequency sub-image corresponding to the first image, and at least one second low-frequency sub-image and at least one second high-frequency sub-image corresponding to the second image, wherein the non-subsampled contourlet transform has translation invariance, and even if the first image is subjected to noise processing, the low-frequency and high-frequency features corresponding to the first image can still be extracted, and the non-subsampled contourlet transform has multi-resolution and multi-directionality, and for the first image and the second image with different resolutions, features in multiple directions can still be extracted based on different directions for constructing a dictionary.

[0043] In an application scenario, a layer of four-directional NSCT transformation is performed on the first image and the second image in each initial image group to obtain five sub-images corresponding to the first image and five sub-images corresponding to the second image, wherein the five sub-images corresponding to the first image and the second image respectively include a low-frequency sub-image and four high-frequency sub-images.

[0044] S204: Divide the first sub-image and the second sub-image in each training image group into the same number of image blocks to obtain training image block sets corresponding to each training image group.

[0045] Specifically, based on a preset ratio between the pixel sizes corresponding to the first sub-image and the second sub-image in each training image group, the first sub-image and the second sub-image are divided into the same number of image blocks. That is, when the pixel size of the first sub-image is N times that of the second sub-image, the pixel size of the image block corresponding to the first sub-image is N times the pixel size of the image block corresponding to the second sub-image, thereby ensuring that the first sub-image and the second sub-image are divided into the same number of image blocks.

[0046] Furthermore, an image block corresponding to the first sub-image and an image block corresponding to the second sub-image with the same transformation parameters are combined into an image block set to obtain more detailed features in the image block set, and the same training image block set includes image blocks corresponding to high-resolution images and image blocks corresponding to low-resolution images, so that the high-resolution image and the low-resolution image are associated.

[0047] In one application, the first sub-image and the second sub-image in each training image group are segmented based on a preset ratio to obtain an equal number of image blocks corresponding to the first sub-image and image blocks corresponding to the second sub-image; and the image blocks corresponding to the first sub-image and the image blocks corresponding to the second sub-image in each training image group are combined into a training image block set.

[0048] Specifically, based on a preset ratio when the first image is downsampled, the first sub-image and the second sub-image in each training image group are segmented so that the pixel size of the image block corresponding to the first sub-image and the pixel size of the image block corresponding to the second sub-image are in a preset ratio, thereby determining the same number of image blocks corresponding to the first sub-image and image blocks corresponding to the second sub-image, so that the image blocks corresponding to the first sub-image and the image blocks corresponding to the second sub-image can be matched one by one, thereby forming a training image block set. Therefore, the training image block set corresponds to image blocks corresponding to the high-resolution image and image blocks corresponding to the low-resolution image, and the number of the two is the same. When the two match, the high-resolution image block and the low-resolution image block can be associated, thereby facilitating the subsequent determination of the first dictionary corresponding to the high-resolution image and the second dictionary corresponding to the low-resolution image.

[0049] S205: In the training image block set corresponding to the same transformation parameter, the image block corresponding to the first sub-image and the image block corresponding to the second sub-image are converted into one-dimensional vectors according to pixel size and concatenated to obtain an initial vector.

[0050] Specifically, based on the pixel size corresponding to the first sub-image and the pixel size corresponding to the second sub-image, in a set of training image blocks corresponding to the same transformation parameter, the image blocks are converted into one-dimensional vectors and concatenated in sequence to obtain an initial vector.

[0051] In one application scenario, the preset ratio between the pixel size corresponding to the first sub-image and the pixel size corresponding to the second sub-image is 2, the image block corresponding to the first sub-image is divided into multiple image blocks according to 18x18, and the image block corresponding to the second sub-image is divided into multiple image blocks according to 9x9. The image block corresponding to the first sub-image is converted into a one-dimensional vector, the image block corresponding to the second sub-image is converted into a one-dimensional vector, and the two one-dimensional vectors are spliced ​​in sequence to obtain an initial vector of 405x1, wherein the splicing order of the image blocks in the initial vector can be defined and selected.

[0052] S206: Convert the initial vector into a multi-dimensional vector to obtain a joint dictionary corresponding to the training image block set.

[0053] Specifically, the initial vector is converted into a multi-dimensional vector to make the initial vector more sparse, thereby obtaining a joint dictionary corresponding to the training image block set.

[0054] In one application scenario, an initial vector of 405x1 is obtained, and the initial vector is converted into a vector of 405x1024 dimension, thereby obtaining a joint dictionary, so that the atoms in the joint dictionary are more sparse. In other application scenarios, the dimension of the joint dictionary can also be customized, and this application does not make specific restrictions on this.

[0055] S207: Determine sparse coefficients in the joint dictionary based on an orthogonal matching pursuit algorithm.

[0056] Specifically, the sparse coefficients corresponding to the joint dictionary are determined based on the orthogonal matching pursuit (OMP) algorithm, where the sparse coefficients are a matrix, and the sparse coefficients are multiplied by the joint dictionary to feedback the features corresponding to the initial vector, and the sparse coefficients have fewer non-zero values ​​in their elements. The joint dictionary is a dimensionality reduction representation of a large data set, and the joint dictionary includes the better features corresponding to the image. Through the matched sparse coefficients, as many features as possible can be represented with as few resources as possible, so that the efficiency is also higher when performing super-resolution reconstruction.

[0057] S208: Determine a reconstruction loss corresponding to the joint dictionary based on the initial vector, the joint dictionary and the sparse coefficient.

[0058] Specifically, the joint dictionary is multiplied by the sparse coefficient to obtain a first product, and the reconstruction loss corresponding to the joint dictionary is determined based on the difference between the initial vector and the first product and the sparse coefficient. The above process is expressed by the following formula:

[0059]

[0060] Among them, Y is the initial vector, D is the joint dictionary, and X is the sparse coefficient.

[0061] S209: updating the atoms in the joint dictionary based on the singular value decomposition algorithm, and updating the sparse coefficients in the joint dictionary.

[0062] Specifically, the sparse coefficients are fixed first, and the atoms in the joint dictionary are updated column by column based on the singular value decomposition algorithm. The sparse coefficients are updated when the joint dictionary is updated to reduce the reconstruction loss during the updating process.

[0063] In an application scenario, a reconstruction column is extracted from a joint dictionary, and a first element matching the number of columns of the reconstruction column is extracted from the sparse coefficients; wherein the reconstruction column is sequentially extracted from the first column in the joint dictionary; based on the initial vector, the reserved column, and the second element matching the number of columns of the reserved column, the matrix to be decomposed is determined; wherein the reserved column is the other columns in the joint dictionary except the reconstruction column, and the second element is the other elements in the sparse coefficient except the first element; the matrix to be decomposed is decomposed based on the singular value decomposition algorithm to obtain a left singular value matrix and a right singular value matrix, and the first column vector of the left singular value matrix is ​​used to update the reconstruction column, and the first element of the right singular value matrix is ​​updated by the product of the first row vector and the first singular value. The above process is expressed by the formula as follows:

[0064]

[0065] Among them, Y is the initial vector, D is the joint dictionary, X is the sparse coefficient, E k is the matrix to be decomposed, d k To reconstruct the column, d j To retain the column, x k is the first element, x j is the second element.

[0066] Specifically, extract the elements in each column starting from the first column of the joint dictionary, use the extracted columns as the reconstructed columns and the unextracted columns as the reserved columns, first fix the sparse coefficient X, and then update the dictionary column by column. When updating the kth column, record d k is the kth column of dictionary D and d k To reconstruct the column, extract the element that matches the reconstructed column from the sparse coefficients, and record xk as the kth element of the sparse coefficient X, that is, the first element, where E k Based on the initial vector and the second element that matches the reserved column and the number of columns in the reserved column, the reserved column is multiplied with the second element corresponding to the reserved column to obtain the second product, and the second product is subtracted from the initial vector to determine the matrix to be decomposed. The matrix to be decomposed is decomposed based on the singular value decomposition algorithm to obtain E k =U∑V T , take the first column vector of the left singular value matrix U as d k Update the reconstruction column of the joint dictionary and take the right singular value matrix V T The product of the first row vector and the first singular value is taken as xk to update the sparse coefficients.

[0067] Furthermore, a new reconstruction loss is obtained, and the atoms in the joint dictionary and the elements in the sparse coefficients are updated. When the reconstruction loss decreases, the non-zero values ​​in the sparse coefficients are further reduced, thereby improving the sparse representation of the image and improving the efficiency and reconstruction effect of the image reconstruction based on the first dictionary and the second dictionary.

[0068] S210: In response to the reconstruction loss being lower than a preset threshold, terminating the iterative update of the joint dictionary and the sparse coefficients, and decomposing the joint dictionary into a first dictionary and a second dictionary.

[0069] Specifically, when the reconstruction loss is lower than a preset threshold, the iterative update of the joint dictionary and the sparse coefficients is terminated, and the joint dictionary is split in the order of one-dimensional vector splicing to obtain a first dictionary corresponding to the high-resolution image and a second dictionary corresponding to the low-resolution image.

[0070] In one application, the step of decomposing the joint dictionary into a first dictionary and a second dictionary includes: decomposing the joint dictionary based on the dimensions and splicing order of the one-dimensional vectors corresponding to the image blocks corresponding to the first sub-image and the image blocks corresponding to the second sub-image, so as to obtain the first dictionary and the second dictionary.

[0071] Specifically, a first dimension of a one-dimensional vector converted into an image block corresponding to the first sub-image and a second dimension of a one-dimensional vector converted into an image block corresponding to the second sub-image are determined; when the one-dimensional vector converted into the image block corresponding to the first sub-image is in front, the elements of the first dimension in the joint dictionary are used as the first dictionary, and the elements of the second dimension in the joint dictionary are used as the second dictionary; when the one-dimensional vector converted into the image block corresponding to the first sub-image is in the back, the elements of the second dimension in the joint dictionary are used as the second dictionary, and the elements of the first dimension in the joint dictionary are used as the first dictionary, so that the first dictionary corresponds to the high-resolution image and the second dictionary corresponds to the low-resolution image, and the first dictionary and the second dictionary have the same sparse coefficients, so that the low-resolution image can be super-reconstructed based on the first dictionary and the second dictionary after solving the sparse coefficients, thereby improving the quality of the low-resolution image.

[0072] In this embodiment, NSCT transformation with different transformation parameters is performed on the first image above the resolution threshold and the second image below the resolution threshold, and finally converted into image blocks corresponding to the high-resolution image and image blocks corresponding to the low-resolution image to obtain a training image block set, the image blocks corresponding to the high-resolution image and the image blocks corresponding to the low-resolution image are converted into one-dimensional vectors and spliced ​​to obtain an initial vector, the initial vector is sparsely converted to obtain a joint dictionary, the sparse coefficients corresponding to the joint dictionary are determined based on an orthogonal matching pursuit algorithm, and the joint dictionary and the sparse coefficients are updated and optimized using a singular value decomposition algorithm, so that the reconstruction loss determined based on the initial vector, the joint dictionary and the sparse coefficients converges, so that the non-zero values ​​in the sparse coefficients are reduced, thereby improving the sparse representation of the image, and improving the efficiency and reconstruction effect of image reconstruction based on the first dictionary and the second dictionary.

[0073] See also Figure 3 , Figure 3: is a flow chart of an embodiment of an image reconstruction method of the present application, the method comprising:

[0074] S301: Obtain an image to be reconstructed whose resolution is lower than a resolution threshold.

[0075] Specifically, an image with a resolution lower than a resolution threshold is acquired as the image to be reconstructed.

[0076] In an application scenario, a remote sensing image taken by a remote sensing camera is acquired, and an image with a resolution lower than a resolution threshold in the remote sensing image is used as an image to be reconstructed.

[0077] S302: Performing non-subsampled contourlet transform on the image to be reconstructed according to different transformation parameters to obtain a plurality of sub-images to be reconstructed corresponding to the image to be reconstructed.

[0078] Specifically, the image to be reconstructed is subjected to NSCT transformation according to different transformation parameters to obtain a plurality of sub-images to be reconstructed corresponding to the image to be reconstructed, wherein the transformation parameters include the number of transformation layers and the transformation direction.

[0079] In one application, NSCT transformation is performed on the image to be reconstructed according to the corresponding transformation parameters when the first dictionary and the second dictionary are generated in the above-mentioned embodiment, and non-subsampled contourlet transformation is performed on the image to be reconstructed according to different transformation layers and transformation directions to obtain at least one low-frequency sub-image to be reconstructed and at least one high-frequency sub-image to be reconstructed corresponding to the image to be reconstructed, as multiple sub-images to be reconstructed.

[0080] S303: Divide the sub-image to be reconstructed corresponding to each transformation parameter into a plurality of image blocks to be reconstructed.

[0081] Specifically, based on the pixel size corresponding to the sub-image to be reconstructed, the sub-image to be reconstructed corresponding to each transformation parameter is divided into a plurality of image blocks.

[0082] In one application, based on the pixel size of the sub-image to be reconstructed corresponding to each transformation parameter, the sub-image to be reconstructed is divided into a plurality of image blocks to be reconstructed, and every two adjacent image blocks to be reconstructed at least partially overlap.

[0083] Specifically, the sub-image to be reconstructed is adaptively segmented according to the pixel size corresponding to the sub-image to be reconstructed, and during segmentation, it is ensured that in the image block to be reconstructed corresponding to the image to be reconstructed, every two adjacent image blocks to be reconstructed include at least a partially overlapping area, so as to avoid discontinuity between the image blocks to be reconstructed, and then when reconstructing the image, after the image blocks are spliced, the reconstructed image is more natural and has a higher effect. Among them, the overlapping position corresponding to the image blocks at the edge position of the image to be reconstructed is dynamically adjusted, and the pixel size of the overlapping area corresponding to the image blocks at the non-edge position of the image to be reconstructed is the same.

[0084] In one application scenario, the image block to be reconstructed is divided into 9×9 blocks according to the pixel size, and each image block to be reconstructed has a 3×3 overlap with the adjacent image block to be reconstructed. For image blocks that are not integer multiples, the overlap size of the image block to be reconstructed at the outermost position is dynamically adjusted.

[0085] S304: Convert the image block to be reconstructed into a one-dimensional vector, and determine the sparse coefficient corresponding to each one-dimensional vector in the second dictionary corresponding to each one-dimensional vector based on the orthogonal matching pursuit algorithm.

[0086] Specifically, the image block to be reconstructed is converted into a one-dimensional vector according to the pixel size of the image block to be reconstructed, and the sparse coefficient corresponding to the one-dimensional vector is calculated in a second dictionary corresponding to the one-dimensional vector based on an orthogonal matching pursuit algorithm.

[0087] In one application scenario, the kth image block to be reconstructed is marked as The sparse coefficients of the image block to be reconstructed in the second dictionary are marked as In the initial stage, the residual r 0 for The index set A is empty, find the residual r k The subscript with the largest inner product with each atom in the second dictionary is denoted by d i is the atom in the second dictionary, N is the dimension of the second dictionary, and the most relevant atom index found is added to the index set, i.e. A k+1 =A k ∪{λ k+1}, thereby updating the residual to obtain When the residual is less than the threshold, the iteration ends and the sparse coefficient is Among them, the sparse coefficients corresponding to the first dictionary and the second dictionary corresponding to the same transformation parameter are the same, and the sparse coefficients have fewer non-zero values, so that the sparse coefficients meet a certain sparsity, in order to solve the ill-conditioned problem of low-dimensional to high-dimensional transformation, and when the sparse coefficients have fewer non-zero values, the efficiency of image reconstruction is higher and the reconstruction effect is better.

[0088] S305: Based on the first dictionary corresponding to each one-dimensional vector and the sparse coefficient corresponding to each one-dimensional vector, generate a reconstructed image block corresponding to each image block to be reconstructed, and splice each reconstructed image block in the segmentation order to generate a reconstructed image corresponding to the image to be reconstructed.

[0089] Specifically, the first dictionary and the second dictionary corresponding to the one-dimensional vector match the transformation parameters corresponding to the sub-image to be reconstructed corresponding to the one-dimensional vector, and the first dictionary and the second dictionary corresponding to each transformation parameter are obtained based on the method described in any of the above embodiments.

[0090] In one application, the sparse coefficient corresponding to each one-dimensional vector is multiplied by the first dictionary corresponding to each one-dimensional vector to obtain a reconstruction vector corresponding to each image block to be reconstructed; the reconstruction vector corresponding to each image block to be reconstructed is converted into a reconstructed image block; all the reconstructed image blocks corresponding to each sub-image to be reconstructed are spliced ​​in a segmentation order to obtain a reconstructed sub-image corresponding to each sub-image to be reconstructed; wherein the overlapping parts between every two adjacent reconstructed image blocks in the reconstructed sub-image are averaged during splicing; and a reconstructed image corresponding to the image to be reconstructed is generated based on each reconstructed sub-image.

[0091] Specifically, the sparse coefficient corresponding to each one-dimensional vector is multiplied by the first dictionary corresponding to each one-dimensional vector, so as to determine the reconstruction vector corresponding to the image block to be reconstructed, the reconstruction vector corresponds to the dimension of the high-resolution image block, the reconstruction vector corresponding to each image block to be reconstructed is converted into a reconstructed image block to obtain a high-resolution image block, the reconstructed image blocks are spliced ​​in the order of segmentation to obtain reconstructed sub-images corresponding to each sub-image to be reconstructed, when the reconstructed image blocks are spliced, the overlapping parts between the image blocks to be reconstructed corresponding to the reconstructed image blocks also include the overlapping parts on the reconstructed image blocks, and the overlapping parts are enlarged according to a preset ratio, the overlapping parts between the reconstructed image blocks are averaged, so as to remove the grid effect of the overlapping parts and improve the continuity of the reconstructed sub-images, based on the reconstructed sub-images corresponding to the multiple sub-images to be reconstructed generated during the NSCT transformation, the reconstructed sub-images are fused to obtain the reconstructed image corresponding to the image to be reconstructed, so as to complete the super-resolution reconstruction of the low-resolution image based on the first dictionary and the second dictionary.

[0092] The above scheme performs a non-subsampled contourlet transform on the image to be reconstructed, thereby obtaining multiple sub-images to be reconstructed, dividing the sub-image to be reconstructed into multiple image blocks to be reconstructed, converting the image blocks to be reconstructed into one-dimensional vectors, solving sparse coefficients in a second dictionary corresponding to the one-dimensional vector, and ensuring that the sparse coefficients are sparse, based on the first dictionary and the second dictionary corresponding to the same transformation parameter share the same sparse coefficients, based on the sparse coefficients corresponding to the image blocks to be reconstructed and the first dictionary, converting the image blocks to be reconstructed into high-resolution reconstructed image blocks, splicing the reconstructed image blocks to obtain reconstructed sub-images, and generating reconstructed images with a resolution exceeding a resolution threshold based on the reconstructed sub-images corresponding to the respective transformation parameters, thereby improving the efficiency and reconstruction effect of low-resolution image reconstruction.

[0093] See also Figure 4 , Figure 4It is a structural diagram of an embodiment of an electronic device of the present application, wherein the electronic device 40 includes a memory 401 and a processor 402 coupled to each other, wherein the memory 401 stores program data (not shown), and the processor 402 calls the program data to implement the method in any of the above embodiments. For descriptions of related contents, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0094] See also Figure 5 , Figure 5 It is a structural diagram of an implementation method of a computer storage medium of the present application. The computer storage medium 50 stores program data 500. When the program data 500 is executed by a processor, the method in any of the above embodiments is implemented. For an explanation of the relevant content, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0095] It should be noted that the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present implementation scheme.

[0096] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0097] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0098] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for generating an image dictionary, characterized in that: The method comprises: Obtaining a plurality of initial image groups; wherein each of the initial image groups comprises a first image having a resolution exceeding a resolution threshold, and a second image having a resolution lower than the resolution threshold generated based on the first image; The first image and the second image in each of the initial image groups are respectively subjected to non-subsampled contourlet transform according to different transformation parameters to obtain a plurality of first sub-images corresponding to the first image and a plurality of second sub-images corresponding to the second image; specifically comprising: performing non-subsampled contourlet transform on the first image in each of the initial image groups according to different transformation levels and transformation directions to obtain at least one first low-frequency sub-image and at least one first high-frequency sub-image corresponding to the first image as the plurality of first sub-images; and performing non-subsampled contourlet transform on the second image in each of the initial image groups according to different transformation levels and transformation directions to obtain at least one second low-frequency sub-image and at least one second high-frequency sub-image corresponding to the second image as the plurality of second sub-images; wherein the transformation parameters include the transformation levels and the transformation directions; Constructing a plurality of training image groups based on a plurality of the first sub-images and a plurality of the second sub-images; wherein the same training image group includes the first sub-images and the second sub-images having the same transformation parameters in the same initial image group; Based on each of the training image groups, a first dictionary and a second dictionary corresponding to each of the transformation parameters are generated; wherein the first dictionary corresponds to images with a resolution higher than a resolution threshold, and the second dictionary corresponds to images with a resolution lower than the resolution threshold.

2. The method for generating an image dictionary according to claim 1, characterized in that: Before the step of generating the first dictionary and the second dictionary corresponding to each of the transformation parameters based on each of the training image groups, the method further includes: The first sub-image and the second sub-image in each of the training image groups are divided into the same number of image blocks to obtain training image block sets corresponding to each of the training image groups.

3. The method for generating an image dictionary according to claim 2, characterized in that: The step of generating the first dictionary and the second dictionary corresponding to each of the transformation parameters respectively based on each of the training image groups comprises: In the training image block set corresponding to the same transformation parameter, the image block corresponding to the first sub-image and the image block corresponding to the second sub-image are converted into one-dimensional vectors according to pixel size and spliced ​​to obtain an initial vector; Converting the initial vector into a multi-dimensional vector to obtain a joint dictionary corresponding to the training image block set; determining sparse coefficients in the joint dictionary based on an orthogonal matching pursuit algorithm; Determine a reconstruction loss corresponding to the joint dictionary based on the initial vector, the joint dictionary and the sparse coefficient; updating the atoms in the joint dictionary based on a singular value decomposition algorithm in the joint dictionary, and updating the sparse coefficients; In response to the reconstruction loss being lower than a preset threshold, the iterative update of the joint dictionary and the sparse coefficients is terminated, and the joint dictionary is decomposed into the first dictionary and the second dictionary.

4. The method for generating an image dictionary according to claim 3, characterized in that: The step of updating the atoms in the joint dictionary based on the singular value decomposition algorithm and updating the sparse coefficients in the joint dictionary includes: Extracting a reconstructed column from the joint dictionary, and extracting a first element matching the number of columns of the reconstructed column from the sparse coefficients; wherein the reconstructed column is sequentially extracted from the first column in the joint dictionary; Determine a matrix to be decomposed based on the initial vector, the reserved columns and a second element matching the number of columns of the reserved columns; wherein the reserved columns are other columns in the joint dictionary except the reconstructed columns, and the second elements are other elements in the sparse coefficients except the first elements; The matrix to be decomposed is decomposed based on a singular value decomposition algorithm to obtain a left singular value matrix and a right singular value matrix, the reconstructed column is updated using the first column vector of the left singular value matrix, and the first element is updated using the product of the first row vector of the right singular value matrix and the first singular value.

5. The method for generating an image dictionary according to claim 3, characterized in that: The step of decomposing the joint dictionary into the first dictionary and the second dictionary comprises: The joint dictionary is decomposed based on the dimensions and splicing order of the one-dimensional vectors corresponding to the image blocks corresponding to the first sub-image and the image blocks corresponding to the second sub-image, so as to obtain the first dictionary and the second dictionary.

6. The method for generating an image dictionary according to claim 2, characterized in that: The step of obtaining a plurality of initial image groups comprises: Obtaining a plurality of first images with resolutions exceeding the resolution threshold, and performing noise addition processing on each of the first images; Down-sampling each of the first images after noise addition processing is performed to obtain the second images corresponding to each of the first images, whose resolution is lower than the resolution threshold, so as to generate a plurality of the initial image groups, and the ratio between the pixel sizes of the first image and the second image is a preset ratio.

7. The method for generating an image dictionary according to claim 6, characterized in that: The step of dividing the first sub-image and the second sub-image in each of the training image groups into the same number of image blocks to obtain a set of training image blocks corresponding to each of the training image groups comprises: Segmenting the first sub-image and the second sub-image in each of the training image groups based on the preset ratio to obtain the same number of image blocks corresponding to the first sub-image and image blocks corresponding to the second sub-image; The image blocks corresponding to the first sub-image and the image blocks corresponding to the second sub-image in each of the training image groups are combined into the training image block set.

8. An image reconstruction method, characterized in that: The method comprises: Obtaining an image to be reconstructed with a resolution lower than a resolution threshold; Performing non-subsampled contourlet transform on the image to be reconstructed according to different transform parameters to obtain a plurality of sub-images to be reconstructed corresponding to the image to be reconstructed; Dividing the sub-image to be reconstructed corresponding to each transformation parameter into a plurality of image blocks to be reconstructed; Convert the image block to be reconstructed into a one-dimensional vector, and determine the sparse coefficient corresponding to each of the one-dimensional vectors in a second dictionary corresponding to each of the one-dimensional vectors based on an orthogonal matching pursuit algorithm; Based on the first dictionary corresponding to each of the one-dimensional vectors and the sparse coefficients corresponding to each of the one-dimensional vectors, a reconstructed image block corresponding to each of the image blocks to be reconstructed is generated, and each of the reconstructed image blocks is spliced ​​in a segmentation order to generate a reconstructed image corresponding to the image to be reconstructed; The first dictionary and the second dictionary corresponding to the one-dimensional vector match the transformation parameters corresponding to the sub-image to be reconstructed corresponding to the one-dimensional vector, and the first dictionary and the second dictionary corresponding to each of the transformation parameters are obtained based on the method described in any one of claims 1 to 7.

9. The image reconstruction method according to claim 8, characterized in that: The step of dividing the sub-image to be reconstructed corresponding to each transformation parameter into a plurality of image blocks to be reconstructed comprises: Based on the pixel size of the sub-image to be reconstructed corresponding to each of the transformation parameters, the sub-image to be reconstructed is divided into a plurality of image blocks to be reconstructed, and every two adjacent image blocks to be reconstructed at least partially overlap.

10. The image reconstruction method according to claim 9, characterized in that: The step of generating reconstructed image blocks corresponding to each of the image blocks to be reconstructed based on the first dictionary corresponding to each of the one-dimensional vectors and the sparse coefficients corresponding to each of the one-dimensional vectors, and splicing each of the reconstructed image blocks in a segmentation order to generate a reconstructed image corresponding to the image to be reconstructed comprises: Multiplying the sparse coefficient corresponding to each of the one-dimensional vectors with the first dictionary corresponding to each of the one-dimensional vectors to obtain a reconstruction vector corresponding to each of the image blocks to be reconstructed; Converting the reconstruction vectors corresponding to the image blocks to be reconstructed into the reconstructed image blocks; All the reconstructed image blocks corresponding to the sub-images to be reconstructed are spliced ​​in the segmentation order to obtain reconstructed sub-images corresponding to the sub-images to be reconstructed; wherein the overlapped parts between every two adjacent reconstructed image blocks in the reconstructed sub-images are averaged when splicing; A reconstructed image corresponding to the image to be reconstructed is generated based on each of the reconstructed sub-images.

11. An electronic device, characterized in that: include: A memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method as claimed in any one of claims 1-7 or 8-10.

12. A computer-readable storage medium having program data stored thereon, characterized in that: When the program data is executed by a processor, the method according to any one of claims 1 to 7 or 8 to 10 is implemented.

Images