An unsupervised implicit modeling blind super-resolution reconstruction method and device

Abstract

This invention relates to an unsupervised implicit modeling blind super-resolution reconstruction method and apparatus. The method includes: S101 acquiring a first low-resolution image; S102 constructing and outputting a pseudo-high-resolution image from the first low-resolution image; S103 inputting the currently obtained pseudo-high-resolution image into a linear convolutional network to train a target degradation network; S104 outputting the target degradation network and a second low-resolution image, wherein the second low-resolution image is obtained by degradation from the first low-resolution image; S105 fixing the target degradation network and inputting the first and second low-resolution images into a convolutional neural network to train the target super-resolution network; S106 calculating the total training loss using standard super-resolution loss and patch similarity to evaluate the target super-resolution network; S107 determining whether S106 has converged, if so, training ends; otherwise, returning to S103. This method can generate accurate and textured reconstructed images.

Description

[0001] Priority application

[0002] This application claims priority to Chinese invention patent application CN2023101816355, filed on February 28, 2023, entitled “An Unsupervised Implicit Modeling Blind Super-Resolution Reconstruction Method and Apparatus”, which is incorporated herein by reference in its entirety. Technical Field

[0003] This invention relates to the field of image super-resolution technology, and in particular to an unsupervised implicit modeling blind super-resolution reconstruction method and apparatus. Background Technology

[0004] The goal of blind super-resolution algorithms is to generate realistic high-resolution images from low-resolution images when the degradation type is unknown. However, existing blind super-resolution algorithms still have some problems when applied to various complex real images. For example, when a super-resolution network trained on training data with predefined degradation types is directly applied to real images, significant domain differences (e.g., the predefined degradation type does not match the degradation type of the real image, or the data distribution of the training data does not match the real data) lead to very low quality reconstructed images.

[0005] For example, patent application CN114202459A discloses a blind image super-resolution method based on depth prior. This method uses a deep convolutional neural network (DIP-Net) combined with non-local attention to estimate high-resolution images; it estimates the blur kernel by solving an exact solution to an optimization problem regarding the blur kernel; it iteratively updates the blur kernel and the high-resolution image, uses the estimated blur kernel to generate a downsampled image of the low-resolution image, and uses the loss function to update the network parameters using the low-resolution image. However, when this method is applied to complex real-world images (e.g., medical images), it is prone to rapid convergence, making it difficult to effectively perform deep learning on the images during training, resulting in poor stability and accuracy of the final reconstruction model.

[0006] For example, patent application CN115578263A discloses a CT super-resolution reconstruction method, system, and device based on generative networks. This method adjusts a CT reconstruction network that is unsuitable for a particular patient to one suitable for that patient's condition without using a large dataset for training. However, from a practical application perspective, custom modeling for a specific patient is both time-consuming and labor-intensive. Summary of the Invention

[0007] The purpose of this invention is to provide an unsupervised implicit modeling blind super-resolution reconstruction method, which partially solves or alleviates the above-mentioned shortcomings in the prior art and can effectively improve the accuracy of high-resolution image reconstruction.

[0008] To solve the aforementioned technical problems, the present invention specifically adopts the following technical solution:

[0009] An unsupervised implicit modeling blind super-resolution reconstruction method includes the following steps:

[0010] S101 acquires the first low-resolution image;

[0011] S102 constructs the first low-resolution image using a super-resolution network construction method and outputs the first pseudo-high-resolution image;

[0012] S103 acquires the currently output pseudo-high resolution image and inputs the pseudo-high resolution image into a linear convolutional network to train the target degradation network;

[0013] S104 outputs the target degradation network obtained in S103 and the second low-resolution image, wherein the second low-resolution image is obtained by the target degradation network degrading the first low-resolution image;

[0014] S105 fixes the target degradation network output by S104, and inputs the first low-resolution image and the currently output second low-resolution image into a convolutional neural network to train a target super-resolution network; S106 evaluates the target super-resolution network trained in S105, wherein...

[0015] S106 includes:

[0016] S106a calculates the standard super-resolution loss L generated by the first low-resolution image and the second pseudo-high-resolution image using the super-resolution loss function. SR The second pseudo-high resolution image is obtained by super-resolution of the second low resolution image using the currently trained target super-resolution network.

[0017] S106b calculates the patch similarity L between the first low-resolution image and the third pseudo-high-resolution image using a patch loss function. GAN The third pseudo-high resolution image is obtained by super-resolution of the first low resolution image using the currently trained target super-resolution network.

[0018] S106c calculates the total training loss generated in S106a and S106b using the total loss function;

[0019] S107 determines whether step S106 has converged. If yes, training ends and the current target super-resolution network is output. If no, the first low-resolution image is constructed using the currently trained target super-resolution network to obtain a new pseudo-high-resolution image, and the new pseudo-high-resolution image is input into S103.

[0020] In some embodiments, the method further includes the following steps prior to S102:

[0021] The first low-resolution image is segmented to obtain several test blocks;

[0022] Calculate the similarity index among the plurality of image patches to be tested. When the similarity index falls within a preset similarity threshold range, input the first low-resolution image into S102.

[0023] In some embodiments, the total loss function is:

[0024] ;

[0025] in, The total training loss is... As weight.

[0026] In some embodiments, S106a includes:

[0027] The first low-resolution image and the second pseudo-high-resolution image are respectively segmented to obtain a number of first image patches and a number of second image patches, wherein the size of the image patch is: 60 pixels * 60 pixels - 120 pixels * 120 pixels;

[0028] The super-resolution loss function is used to calculate the pixel information of the first and second image patches to determine the standard super-resolution loss; wherein the super-resolution loss function is a mean square loss function.

[0029] In some embodiments, S106b includes the step of:

[0030] The first low-resolution image and the third pseudo-high-resolution image are respectively segmented to obtain several first patches and second patches accordingly;

[0031] Obtain feature information of the first patch and the second patch, and calculate the patch similarity L between the first low-resolution image and the third pseudo-high-resolution image using the feature information. GAN .

[0032] In some embodiments, the model for calculating the patch similarity is:

[0033] ;

[0034] ;

[0035] ;

[0036] ;

[0037] ;

[0038] Where x and y represent the texture features that differ in the first patch and the second patch, respectively, avg represents the average intra-class distance between the two types of features, C represents the set of the two types of features, U represents the size of the feature center point, dcen represents the distance between the center points of the two types of features, and FCD represents the clustering distance between the two types of features.

[0039] In some embodiments, the patch size is 8*8 pixels to 13*13 pixels.

[0040] In some embodiments, the training objective in S103 is: to minimize ;

[0041] and ;

[0042] in, Here, MSE is the training loss function for the degenerate network, and the mean squared loss function is... The pixel information of the first low-resolution image. This is the super-resolution network obtained through current training. This is the degenerate network obtained from the current training.

[0043] In some embodiments, in S103, the linear convolutional network includes:

[0044] ;

[0045] ;

[0046] ;

[0047] ;

[0048] ;

[0049] in, This represents the input of the i-th hidden layer. This represents the grayscale value of a pixel in a pseudo-high-resolution image. This represents the i-th convolutional layer. Indicates the number of convolutional layers. This represents an equivalent large convolutional layer obtained by convolving n convolutional layers, where * represents a convolution operation. This represents the image obtained by sampling a pseudo-high-resolution image. This represents the output downsampled image.

[0050] The present invention also provides an unsupervised implicit modeling blind super-resolution reconstruction device, comprising:

[0051] The sample acquisition module is configured to acquire a first low-resolution image;

[0052] The image reconstruction module is configured to construct the first low-resolution image using a super-resolution network construction method and output a first pseudo-high-resolution image.

[0053] The degradation network training module is configured to acquire a pseudo-high resolution image of the current output and input the pseudo-high resolution image into a linear convolutional network to train the target degradation network.

[0054] The output module is configured to output the target degraded network obtained in the degraded network training module and the second low-resolution image, wherein the second low-resolution image is obtained by the target degraded network degrading the first low-resolution image;

[0055] A super-resolution network training module is configured to fix the target degradation network output by the output module, and input the first low-resolution image and the currently output second low-resolution image into a convolutional neural network to train the target super-resolution network; an evaluation module is configured to evaluate the target super-resolution network trained in the super-resolution network training module, wherein...

[0056] The evaluation module includes:

[0057] The first unit is configured to calculate the standard super-resolution loss L generated by the first low-resolution image and the second pseudo-high-resolution image using a super-resolution loss function. SR The second pseudo-high resolution image is obtained by super-resolution of the second low resolution image using the currently trained target super-resolution network.

[0058] The second unit is configured to calculate the patch similarity L between the first low-resolution image and the third pseudo-high-resolution image using a patch loss function. GAN The third pseudo-high resolution image is obtained by super-resolution of the first low resolution image using the currently trained target super-resolution network.

[0059] The third unit is configured to calculate the total training loss generated in the first and second units using the total loss function;

[0060] The judgment module determines whether the target super-resolution network in the step has converged. If it has, the training ends and the current target super-resolution network is output. If not, the target super-resolution network trained at the moment is used to construct a new pseudo-high resolution image from the first low-resolution image, and the new pseudo-high resolution image is input into the degradation network training module.

[0061] Beneficial technical effects:

[0062] This invention proposes an unsupervised, iterative, patch-similar-based blind super-resolution reconstruction method. In the construction of the super-resolution network, a dual-branch computation method (i.e., simultaneously calculating the standard super-resolution loss and patch similarity) is used to perform deep learning on a single image, avoiding the problem of rapid convergence during training. Finally, a super-resolution network with good reconstruction quality and stability is obtained. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. The elements or parts in the drawings are not necessarily drawn to scale. Obviously, the drawings described below are some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0064] Figure 1 This is a schematic flowchart of a blind super-resolution reconstruction method in an exemplary embodiment of the present invention;

[0065] Figure 2 This is a schematic diagram of a preferred algorithm in a blind super-resolution reconstruction method according to an exemplary embodiment of the present invention;

[0066] Figure 3a This is a comparison chart showing the results of high-resolution image reconstruction using existing reconstruction methods and the reconstruction method of this invention.

[0067] Figure 3b This refers to the experimental results data of the reconstruction method in this invention and existing reconstruction methods;

[0068] Figure 4 The images show CT images of the foot and ankle, as well as high-resolution images reconstructed from the original CT images using bicubic interpolation, ZSSR, KernelGAN, MZSR, and the method of this invention, respectively.

[0069] Figure 5 The results of reference-free contrast on foot and ankle CT images are shown using various blind super-resolution algorithms.

[0070] Figure 6a The image shows a CT scan of the foot and ankle;

[0071] Figure 6b This diagram illustrates the visualization of similarity in foot and ankle CT images.

[0072] Figure 7 This is a schematic diagram of a reconstruction apparatus in an exemplary embodiment of the present invention. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0074] In this document, suffixes such as "module," "part," or "unit" used to denote elements are used only for the purpose of illustrative purposes and have no specific meaning in themselves. Therefore, "module," "part," or "unit" may be used interchangeably.

[0075] In this document, the terms "upper," "lower," "inner," "outer," "front," "rear," "one end," and "the other end," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the present invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the present invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0076] In this document, unless otherwise explicitly specified and limited, the terms "installed," "equipped with," "connected," etc., should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, a direct connection, or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0077] In this document, "and / or" includes any and all combinations of one or more of the listed related items.

[0078] In this article, "multiple" means two or more, that is, it includes two, three, four, five, etc.

[0079] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0080] As used in this specification, the term "about" typically means + / -5% of the value, more typically + / -4% of the value, more typically + / -3% of the value, more typically + / -2% of the value, even more typically + / -1% of the value, even more typically + / -0.5% of the value.

[0081] In this specification, some embodiments may be disclosed in a range-bound format. It should be understood that this "range-bound" description is merely for convenience and brevity and should not be construed as a rigid limitation on the disclosed range. Therefore, the description of a range should be considered as having specifically disclosed all possible subranges and the individual numerical values ​​within that range. For example, a description of the range 1-6 should be considered as having specifically disclosed subranges such as from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6, etc., and the individual numbers within that range, such as 1, 2, 3, 4, 5, and 6. This rule applies regardless of the breadth of the range.

[0082] Definition of noun:

[0083] In this paper, "first low-resolution image" refers to the "original low-resolution image" to be reconstructed, or "original image". In embodiments of the present invention, the first low-resolution image generally refers to an image with high similarity (specifically, similarity of texture features) between various local features (or at least some local features in a region).

[0084] For example, the first low-resolution image can be a medical image with highly similar texture features. Specifically, a medical image refers to a test or scan image obtained through various medical imaging techniques such as X-ray imaging, ultrasound imaging, CT imaging, nuclear medicine imaging, and (nuclear) magnetic resonance imaging (MRI).

[0085] Alternatively, the first low-resolution image can be a natural image, such as a portrait or architectural image, that exhibits high similarity in local features (or local regions). For example, the skin texture in the eye area or facial region of a portrait image has high similarity; similarly, the structural design of a building's exterior walls often shows high similarity. Specifically, the first low-resolution image can also be a local image extracted from the original image, such as a local eye image or face image extracted from a natural photograph containing a portrait. Another example is a local architectural image extracted from a landscape photograph.

[0086] In this paper, "pseudo-high resolution image" refers to a high-resolution image obtained by reconstructing a low-resolution image (e.g., the original low-resolution image or a degraded image) using a super-resolution network. In this paper, it is also referred to as "pseudo-high resolution image" or "pseudo-high resolution map".

[0087] In this paper, the "Expectation-Maximization algorithm" is also referred to as the EM algorithm. The EM algorithm is an iterative method for estimating latent parameters and latent variables. The core idea of ​​the EM algorithm is as follows: in the E-step, the latent parameters are fixed, and the optimal latent variables are iteratively inferred from the training data; in the M-step, the latent variables are fixed, and the latent parameters are estimated by maximum likelihood estimation.

[0088] Example 1

[0089] See Figure 1 As shown in Figure 6, this invention provides an unsupervised implicit modeling blind super-resolution reconstruction method, comprising the following steps:

[0090] S101 Acquire the first low-resolution image I test .

[0091] For example, in some embodiments, the first low-resolution image (i.e., the original image) is a CT image, ultrasound, etc.

[0092] S102 uses a super-resolution network construction method to process the first low-resolution image I. test Construct and output the first pseudo-high resolution image SR(I) test );

[0093] For example, in some embodiments, when initially viewing the original image I testWhen performing high-resolution reconstruction, one or more of the following super-resolution networks can be used to construct pseudo-high-resolution images: pre-upsampling super-resolution networks, post-upsampling super-resolution networks, progressive upsampling super-resolution networks, iterative up-and-downsampling super-resolution networks, etc.

[0094] For example, in some embodiments, one or more of the following software or websites may also be used to modify the original I test High-resolution reconstruction: Aiseesoft Image Upscaler Online, RealESRGAN GUI, etc.

[0095] S103 retrieves the currently output pseudo-high resolution image (also known as: pseudo-high resolution image) SR(I test The pseudo-high-resolution image is then input into a linear convolutional network to train a target degradation network.

[0096] S104 outputs the target degradation network F obtained in S103 and the second low-resolution image F(I). test The second low-resolution image is obtained by the target degradation network degrading the first low-resolution image.

[0097] S105 fixes the target degradation network output by S104, and inputs the first low-resolution image and the currently output second low-resolution image into the convolutional neural network to train and obtain the target super-resolution network.

[0098] S106 evaluates the target super-resolution network trained in S105, where...

[0099] S106 includes:

[0100] S106a calculates the standard super-resolution loss L generated by the first low-resolution image and the second pseudo-high-resolution image using the super-resolution loss function. SR The second pseudo-high resolution image is obtained by super-resolution of the second low resolution image using the currently trained target super-resolution network.

[0101] S106b calculates the patch similarity L between the first low-resolution image and the third pseudo-high-resolution image using a patch loss function. GAN The third pseudo-high resolution image is obtained by super-resolution of the first low resolution image using the currently trained target super-resolution network.

[0102] S106c calculates the total training loss generated in S106a and S106b using the total loss function;

[0103] S107 determines whether step S106 has converged. If yes, training ends and the current target super-resolution network is output. If no, the first low-resolution image is constructed using the currently trained target super-resolution network to obtain a new pseudo-high-resolution image, and the new pseudo-high-resolution image is input into S103.

[0104] In some embodiments, when the total training loss calculated in S106 no longer decreases, or when the change trend of the total training loss is very slow (e.g., the value of the currently calculated total training loss is less than or equal to a preset first threshold compared to the value of the previously calculated total training loss), it is determined that S106 has converged, and the currently trained target super-resolution network is output. If S106 has not converged, a new pseudo-height map will be constructed from the original image based on the currently trained target super-resolution network, and the new pseudo-height map will be input into S103. It can be understood that S103-S107 is an iterative training process.

[0105] The first threshold can be set by the user. For example, the user can adjust the size or range of the first threshold according to the accuracy requirements of the super-resolution network.

[0106] In the training process of the super-resolution network, this invention employs a dual-branch computation mode of standard super-resolution loss and texture loss (or patch similarity) to perform deep learning on a single original image, thereby training a target super-resolution network capable of generating high-resolution images with high accuracy (low content loss) and clear texture.

[0107] Furthermore, in some embodiments, to ensure the effectiveness of the training process, a step is included before S102:

[0108] The first low-resolution image (i.e., the original image) is segmented to obtain several test patches;

[0109] Calculate the similarity index (or structural similarity) between the plurality of image patches to be tested. When the similarity index falls within a preset similarity threshold range, the low-resolution image is input into S102.

[0110] In this embodiment of the invention, SSIM (Structural Similarity) is preferably used to evaluate the similarity of local regions (specifically patch regions) in the original image.

[0111] Specifically, in some embodiments, the "similarity index" can be the average of all similarity indices. Here, all similarity indices are a set of similarity index values ​​calculated for each set of comparison images selected from a plurality of test image patches (for example, two spatially adjacent test image patches can be set as a set of comparison images, or any two test image patches can be set as a set of comparison images). In this embodiment, when the "similarity index" falls within a preset similarity threshold range, the original image is considered to have completed effective training and is input into step S102.

[0112] Alternatively, in other embodiments, the "similarity index" can also be represented as a set of all similarity indices. Specifically, in some embodiments, when all similarity indices in the set fall within a preset similarity threshold range, the original image is considered to have completed effective training and is input into step S102. Or, in other embodiments, when the number (or percentage) of similarity indices falling within the similarity threshold range exceeds a preset evaluation metric (e.g., when the percentage of similarity indices falling within the threshold range in the set exceeds 80%, 90%, or other metric settings), the original image is considered to have completed effective training and is input into value S102.

[0113] Specifically, in some embodiments, the size of the patch to be tested is the same as or similar to the size of the patch.

[0114] Specifically, in some embodiments, the similarity threshold range can be adapted by the staff according to the actual application requirements. For example, in one specific embodiment, the similarity threshold range can be [0.65, 1].

[0115] It is understood that this invention is preferably applicable to high-definition reconstruction of original images with high texture similarity, either overall or locally. For example, as Figure 3a As shown, this type of original image can be a person's face or eye image, or it can be a building image.

[0116] For example, such as Figure 6a , Figure 6b As shown, the original image preferably used in this invention is a CT image (such as a foot and ankle CT image).

[0117] Preferably, in some embodiments, before S102, a similarity index of each local region in the original image can be calculated to select a local region suitable for high-definition reconstruction. Specifically, when the texture feature similarity of a certain local region in the original image is high, that local region can be used as a low-resolution image to be reconstructed (e.g., extracting a local building image from a natural image).

[0118] In some embodiments, the total loss function used in S106c is:

[0119] ;

[0120] in, The total training loss is... The weights (which can be adaptively set by the user based on specific training image types or training accuracy requirements, for example, in a specific embodiment) It can be set to 0.01).

[0121] In some embodiments, S106a includes the step of:

[0122] The first low-resolution image and the second pseudo-high-resolution image are respectively segmented to obtain a number of first image patches and a number of second image patches, wherein the size of the image patch is: 60 pixels * 60 pixels - 120 pixels * 120 pixels;

[0123] The super-resolution loss function is used to calculate the pixel information (or pixel-level loss) of the first and second patches to determine the standard super-resolution loss; wherein the super-resolution loss function is a mean square loss function.

[0124] In some embodiments, S106b includes the step of:

[0125] The first low-resolution image and the third pseudo-high-resolution image are respectively segmented to obtain several first patches and second patches accordingly;

[0126] Obtain feature information of the first patch and the second patch, and calculate the patch similarity L between the first low-resolution image and the third pseudo-high-resolution image using the feature information. GAN .

[0127] Preferably, in this embodiment, a generative adversarial network (GAN framework) is selected to measure patch similarity. The GAN framework includes a generator (i.e., a super-resolution network) for minimizing the difference between the original image and the pseudo-high-resolution image (i.e., the image obtained by super-resolution of the original image), and a discriminator for maximizing the difference between the original image and the pseudo-high-resolution image.

[0128] Specifically, in some embodiments, the model for calculating the patch similarity is as follows:

[0129] ;

[0130] ;

[0131] ;

[0132] ;

[0133] ;

[0134] ;

[0135] Where x and y represent the texture features that differ between the first patch and the second patch, respectively (specifically, the texture features here are the differences in details automatically learned by the discriminator in the generative adversarial network, that is, x and y represent the feature data generated by the discriminator through the patches of the original image and the pseudo-high-resolution image, respectively), avg represents the average intra-class distance between the two types of features, C represents the set of the two types of features, U represents the size of the feature center point, dcen represents the distance between the center points of the two types of features, and FCD represents the feature clustering distance (that is, the inter-class distance between the two types of features).

[0136] During the reciprocating cycle of S103-S106, It will gradually converge towards smaller dimensions.

[0137] In this embodiment of the invention, the above-mentioned similarity model is selected to design the discriminator in the GAN framework, which not only ensures the texture accuracy of the super-resolution network during the reconstruction process, but also enhances the stability of the GAN framework during the training process through dimensional consistency.

[0138] Preferably, in some embodiments, the patch size is approximately 8*8 pixels to 13*13 pixels in order to accurately identify the texture details of the patch.

[0139] In this embodiment of the invention, deep learning is applied to small-sized patches to focus attention on the local texture features of the image.

[0140] In some embodiments, the training objective in S103 is: to minimize .

[0141] ;

[0142] in, Here, MSE is the training loss function for the degenerate network, and the mean squared loss function is... The pixel information of the first low-resolution image. This refers to the super-resolution network obtained through training in step S105. This is the degenerate network obtained from the current training of S103.

[0143] Specifically, .in This represents the grayscale value of each pixel in the original image. represents the grayscale value of each pixel in the new degraded image (i.e., the image obtained by super-resolution and degradation of the original image in sequence), and n represents the number of pixels.

[0144] To prevent overfitting with small sample data (in this invention, the training data for the degenerate network comes from a single original image I) test Small sample data), and to avoid the degenerate network learning erroneous information from pseudo-data (the training data uses pseudo-height maps generated by a super-resolution network). In some embodiments, in S103, the linear convolutional network includes:

[0145] ;

[0146] ;

[0147] ;

[0148] ;

[0149] ;

[0150] in, This represents the input of the i-th hidden layer. This represents the grayscale value of a pixel in a pseudo-high-resolution image (i.e., the image reconstructed using the target super-resolution network trained / output). This represents the i-th convolutional layer. Indicates the number of convolutional layers. This represents an equivalent large convolutional layer obtained by convolving n convolutional layers, where * denotes a convolution operation. This represents the image obtained by sampling a pseudo-high-resolution image. This represents the output downsampled image, which is the image obtained after super-resolution (SR) and degradation (F) of the original image.

[0151] Specifically, in some embodiments, n is 7.

[0152] In this embodiment of the invention, by using multi-layer convolutional networks for convolution and normalization processing (equivalent to setting the weights of the blur kernel of the linear convolutional network to 1), the brightness is ensured to be consistent between the pseudo-high-resolution image and the degraded image (i.e. the second low-resolution image) of the learning output, thus preventing S103 from deviating in the wrong direction.

[0153] In some embodiments, the calculation model for the standard super-resolution loss is as follows:

[0154] ;

[0155] Where MSE is the mean square loss function, I testThe image is a low-resolution sample image, SR is a super-resolution network, and F is the target degradation network.

[0156] Specifically, ;

[0157] in, This represents the grayscale value of a pixel in the original image. This represents the grayscale value of a pixel in a pseudo-height map (i.e., an image obtained by successively degrading and super-resolution the original image), where n is the number of pixels.

[0158] In this embodiment of the invention, the GAN (Generative Adversarial Network) framework includes: a generator whose optimization objective is to minimize the similarity of the patches, and a generator whose optimization objective is to maximize the difference between the patches (i.e., The discriminator is the target of optimization. Specifically, the generator is the currently trained or assumed super-resolution network, and the discriminator is the patch loss function.

[0159] In some embodiments, the method further includes the step of:

[0160] Obtain pre-trained samples;

[0161] The super-resolution network and the degradation network are pre-trained using the pre-trained samples.

[0162] For example, in some embodiments, corresponding labels can be added to the pre-training samples, such as adding information on whether there is a lesion and the type of lesion, and a fully connected layer can be added after the super-resolution network is reconstructed, so as to achieve targeted supervised training for a certain type of disease or a certain type of patient.

[0163] Alternatively, in other embodiments, representation learning methods can be used to learn meaningful representational knowledge by constructing a self-supervised dataset, which can help complete subsequent super-resolution tasks.

[0164] Based on the methods described in the above embodiments, this invention also provides an example of an iterative blind super-resolution algorithm (also referred to herein as: pseudocode BSSR). An example of the pseudocode BSSR algorithm is as follows: Figure 2 As shown.

[0165] Furthermore, to verify the reliability of the model training method, experiments were conducted on some real images:

[0166] Experiment 1

[0167] In this experiment, Bicubic interpolation, EDSR (enhanced deep super-resolution network), RCAN (very deep residual channel attention networks), ZSSR, and the method proposed in this invention were used to reconstruct low-resolution images. Figure 3a From left to right, the image shows a low-resolution image (original image), high-resolution images reconstructed using Bicubic, EDSR, RCAN, ZSSR, and the method of this invention, respectively, and a real high-resolution image corresponding to the original image. Figure 3a As can be seen, the method provided by this invention shows good reconstruction results (high accuracy and clear texture) when applied to images with high local texture feature similarity.

[0168] Experiment 2

[0169] This experiment used three super-resolution datasets—Set5 (database 1), BSD100 (database 2), and Urban100 (database 3)—as training data to train the super-resolution network. Figure 3b As shown, this experiment assumes five scenarios, from sampling 1 to 5: specifically, only bicubic downsampling, Isotropic Gaussian blur kernel with width λ=0.2 + direct downsampling. Isotropic Gaussian blur kernel with width λ=2 + direct downsampling Anisotropic Gaussian blur kernel with width λ=1.0 and θ=-0.5 + direct downsampling. Isotropic Gaussian blur kernel with width λ=1.3 + bicubic downsampling. Methods 1-5 are Bicubic, RCAN, IKC (terative kernel correction), ZSSR (ZeroShotSR), and MZSR (zero-shot super-resolution based on meta-transfer learning) algorithms, respectively.

[0170] from Figure 3b As can be seen, the method of this invention exhibits high signal-to-noise ratio and structural similarity under different sampling scenarios. Figure 3b The data in the figure represents the PSNR / SSIM index, which stands for Peak Signal-to-Noise Ratio and Structural Similarity. Higher values ​​for both indicate a greater consistency between the reconstructed high-resolution image and the real high-resolution image.

[0171] Experiment 3

[0172] Furthermore, the present invention also uses CT images as the original images (e.g., Figure 6a As shown, Figure 6b As shown, experiments were conducted. Among them, Figure 6b The similarity visualization of CT image patches is shown. A higher similarity index (represented by a darker color in the visualization) indicates a greater similarity to other patches. Experimental results are as follows... Figures 4-5 As shown, where, Figure 4 The largest image on the far left is the original image. The images on the right, from left to right, are high-resolution images obtained by reconstructing CT images based on bicubic interpolation, ZSSR, KernelGAN, MZSR, and our method (BSSR).

[0173] In this embodiment, slice images from a foot and ankle CT model are used to conduct a comparative experiment on the BSSR algorithm. Since real CT images lack high-resolution ground truth labels for reference, three no-reference super-resolution evaluation metrics were selected as the evaluation criteria: sharpness, Block Self-Similarity (BSSIM), and BRISQUE. The sharpness metric is used to evaluate the sharpness of image edges, BSSIM is used to evaluate the degree of image distortion at different resolutions, and BRISQUE is used to evaluate image texture quality. To avoid the limitations of no-reference evaluation metrics, it is preferable to use all three metrics simultaneously to evaluate algorithm performance.

[0174] Reconstruction results visualization, as shown Figure 4 As shown, it can be seen that the method proposed in this invention can achieve the clearest and best visualization effect. Although the image reconstructed by KernelGAN is clearer than other images, it has over-sharpened artifacts and is no longer realistic.

[0175] Specifically, the results of the three reconstruction indicators are as follows: Figure 5 As shown in the table, BRISQUE is a no-reference image quality metric used to evaluate image quality; a lower metric indicates higher image quality. The table shows that our method performs best on the BRISQUE metric, outperforms most algorithms in sharpness, and has a high block similarity metric. This demonstrates that our method can produce high-resolution images with rich textures, sharp edges, and high quality while maintaining a high degree of similarity to the original image.

[0176] On the other hand, the KernelGAN algorithm achieves the best BRISQUE score (61.33) and sharpness score (12.46) among other algorithms. However, visualization results and block self-similarity analysis show that the reconstructed image from KernelGAN exhibits decreased similarity to the original image, demonstrating the presence of artifacts and oversharpening. This oversharpening effect makes the generated image resemble a realistic CT image, displaying strong artificial traces and altering the key semantics of medical images. This violates the principles of super-resolution reconstruction in medical images, specifically the principle of case invariance.

[0177] The above experiments show that the blind super-resolution reconstruction method provided by this invention can achieve good results in various types of images with high local texture feature similarity (especially medical images).

[0178] Example 2

[0179] like Figure 7 As shown, the present invention also provides an unsupervised implicit modeling blind super-resolution reconstruction device corresponding to the above method, comprising:

[0180] Sample acquisition module 01 is configured to acquire a first low-resolution image;

[0181] Image reconstruction module 02 is configured to construct the first low-resolution image using a super-resolution network construction method and output a first pseudo-high-resolution image;

[0182] The degradation network training module 03 is configured to acquire the pseudo-high resolution image of the current output and input the pseudo-high resolution image into a linear convolutional network to train the target degradation network.

[0183] Output module 04 is configured to output the target degradation network and the second low-resolution image obtained in the degradation network training module, wherein the second low-resolution image is obtained by the target degradation network degrading the first low-resolution image.

[0184] The super-resolution network training module 05 is configured to fix the target degradation network output by the output module, and input the first low-resolution image and the currently output second low-resolution image into the convolutional neural network to train the target super-resolution network.

[0185] The evaluation module is configured to evaluate the target super-resolution network trained in the super-resolution network training module, wherein...

[0186] The evaluation module 06 includes:

[0187] The first unit is configured to calculate the standard super-resolution loss L generated by the first low-resolution image and the second pseudo-high-resolution image using a super-resolution loss function. SR The second pseudo-high resolution image is obtained by super-resolution of the second low resolution image using the currently trained target super-resolution network.

[0188] The second unit is configured to calculate the patch similarity L between the first low-resolution image and the third pseudo-high-resolution image using a patch loss function. GAN The third pseudo-high resolution image is obtained by super-resolution of the first low resolution image using the currently trained target super-resolution network.

[0189] The third unit is configured to calculate the total training loss generated in the first and second units using the total loss function;

[0190] The judgment module determines whether the target super-resolution network in the step has converged. If it has, the training ends and the current target super-resolution network is output. If not, the target super-resolution network trained at the moment is used to construct a new pseudo-high resolution image from the first low-resolution image, and the new pseudo-high resolution image is input into the degradation network training module.

[0191] In some embodiments, the device further includes: an image pre-judgment module, which is configured to segment the first low-resolution image to obtain a plurality of test patches; and calculate a similarity index between the plurality of test patches. When the similarity index is within a preset similarity threshold range, the first low-resolution image is input into an image reconstruction module.

[0192] In some embodiments, the total loss function is:

[0193] ;

[0194] in, The total training loss is... As weight.

[0195] In some embodiments, the first unit is further configured to:

[0196] The first low-resolution image and the second pseudo-high-resolution image are respectively segmented to obtain a number of first image patches and a number of second image patches, wherein the size of the image patch is: 60 pixels * 60 pixels - 120 pixels * 120 pixels;

[0197] The super-resolution loss function is used to calculate the pixel information of the first and second image patches to determine the standard super-resolution loss; wherein the super-resolution loss function is a mean square loss function.

[0198] In some embodiments, the second unit is further configured as follows:

[0199] The first low-resolution image and the third pseudo-high-resolution image are respectively segmented to obtain several first patches and second patches accordingly;

[0200] Obtain feature information of the first patch and the second patch, and calculate the patch similarity L between the first low-resolution image and the third pseudo-high-resolution image using the feature information. GAN .

[0201] In some embodiments, the model for calculating the patch similarity is:

[0202] ;

[0203] ;

[0204] ;

[0205] ;

[0206] ;

[0207] Where x and y represent the texture features that differ in the first patch and the second patch, respectively, avg represents the average intra-class distance between the two types of features, C represents the set of the two types of features, U represents the size of the feature center point, dcen represents the distance between the center points of the two types of features, and FCD represents the clustering distance between the two types of features.

[0208] In some embodiments, the patch size is 8*8 pixels to 13*13 pixels.

[0209] In some embodiments, the training objective in the degenerate network training module is: to minimize ;and ;

[0210] in, Here, MSE is the training loss function for the degenerate network, and the mean squared loss function is... The pixel information of the first low-resolution image. This is the super-resolution network obtained through current training. This is the degenerate network obtained from the current training.

[0211] In some embodiments, the linear convolutional network in the degenerate network training module includes:

[0212] ;

[0213] ;

[0214] ;

[0215] ;

[0216] ;

[0217] in, This represents the input of the i-th hidden layer. This represents the grayscale value of a pixel in a pseudo-high-resolution image. This represents the i-th convolutional layer. Indicates the number of convolutional layers. This represents an equivalent large convolutional layer obtained by convolving n convolutional layers, where * represents a convolution operation. This represents the image obtained by sampling a pseudo-high-resolution image. This represents the output downsampled image.

[0218] It is understood that the unsupervised implicit modeling blind super-resolution reconstruction device in this invention can implement the method steps in any embodiment of this invention, and will not be repeated here.

[0219] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0220] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a computer terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0221] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. An unsupervised implicit modeling blind super-resolution reconstruction method, characterized in that, Including the following steps: S101 acquires the first low-resolution image; S102 constructs the first low-resolution image using a super-resolution network construction method and outputs the first pseudo-high-resolution image; S103 acquires the currently output pseudo-high resolution image and inputs the pseudo-high resolution image into a linear convolutional network to train the target degradation network; S104 outputs the target degradation network obtained in S103 and the second low-resolution image, wherein the second low-resolution image is obtained by the target degradation network degrading the first low-resolution image; S105 fixes the target degradation network output by S104, and inputs the first low-resolution image and the currently output second low-resolution image into a convolutional neural network to train a target super-resolution network; S106 evaluates the target super-resolution network trained in S105, wherein... S106 includes: S106a calculates the standard super-resolution loss L generated by the first low-resolution image and the second pseudo-high-resolution image using the super-resolution loss function. SR The second pseudo-high resolution image is obtained by super-resolution of the second low resolution image using the currently trained target super-resolution network. S106b calculates the patch similarity L between the first low-resolution image and the third pseudo-high-resolution image using a patch loss function. GAN The third pseudo-high resolution image is obtained by super-resolution of the first low resolution image using the currently trained target super-resolution network. S106c calculates the total training loss generated in S106a and S106b using the total loss function; S107 determines whether step S106 has converged. If yes, training ends and the current target super-resolution network is output. If no, the first low-resolution image is constructed using the currently trained target super-resolution network to obtain a new pseudo-high-resolution image, and the new pseudo-high-resolution image is input into S103.

2. The method according to claim 1, characterized in that, Prior to S102, it also includes: The first low-resolution image is segmented to obtain several test blocks; Calculate the similarity index among the plurality of image patches to be tested. When the similarity index falls within a preset similarity threshold range, input the first low-resolution image into S102.

3. The method according to claim 1, characterized in that, The total loss function is: ； in, The total training loss is... As weight.

4. The method according to claim 1, characterized in that, S106a includes: The first low-resolution image and the second pseudo-high-resolution image are respectively segmented to obtain a number of first image patches and a number of second image patches, wherein the size of the image patch is: 60 pixels * 60 pixels - 120 pixels * 120 pixels; The super-resolution loss function is used to calculate the pixel information of the first and second image patches to determine the standard super-resolution loss; wherein the super-resolution loss function is a mean square loss function.

5. The method according to claim 1, characterized in that, S106b includes the following steps: The first low-resolution image and the third pseudo-high-resolution image are respectively segmented to obtain several first patches and second patches accordingly; Obtain feature information of the first patch and the second patch, and calculate the patch similarity L between the first low-resolution image and the third pseudo-high-resolution image using the feature information. GAN .

6. The method according to claim 5, characterized in that, The model for calculating the patch similarity is as follows: ； （1） ； （2） ； （3） ； （4） ； （5） Where x and y represent the texture features that differ between the first patch and the second patch, respectively. avg represents the average intra-class distance between two classes of features, C represents the set of two classes of features, and U represents the distance between features. The size of the centroid, dcen represents the distance between the centroids of the two feature classes, and FCD represents the clustering distance between the two feature classes.

7. The method according to claim 1, characterized in that, The patch size is 8*8 pixels to 13*13 pixels.

8. The method according to claim 1, characterized in that, The training objective in S103 is: to minimize ; and ; in, Here, MSE is the training loss function for the degenerate network, and the mean squared loss function is... The pixel information of the first low-resolution image. This is the super-resolution network obtained through current training. This is the degenerate network obtained from the current training.

9. The method according to claim 8, characterized in that, In S103, the linear convolutional network includes: ； ； ； ； ； in, This represents the input of the i-th hidden layer. This represents the grayscale value of a pixel in a pseudo-high-resolution image. This represents the i-th convolutional layer. Indicates the number of convolutional layers. This represents an equivalent large convolutional layer obtained by convolving n convolutional layers, where * denotes a convolution operation. This represents the image obtained by sampling a pseudo-high-resolution image. This represents the output downsampled image.

10. An unsupervised implicit modeling blind super-resolution reconstruction device, characterized in that, include: The sample acquisition module is configured to acquire a first low-resolution image; The image reconstruction module is configured to construct the first low-resolution image using a super-resolution network construction method and output a first pseudo-high-resolution image. The degradation network training module is configured to acquire a pseudo-high resolution image of the current output and input the pseudo-high resolution image into a linear convolutional network to train the target degradation network. The output module is configured to output the target degraded network obtained in the degraded network training module and the second low-resolution image, wherein the second low-resolution image is obtained by the target degraded network degrading the first low-resolution image; A super-resolution network training module is configured to fix the target degradation network output by the output module, and input the first low-resolution image and the currently output second low-resolution image into a convolutional neural network to train the target super-resolution network; an evaluation module is configured to evaluate the target super-resolution network trained in the super-resolution network training module, wherein... The evaluation module includes: The first unit is configured to calculate the standard super-resolution loss L generated by the first low-resolution image and the second pseudo-high-resolution image using a super-resolution loss function. SR The second pseudo-high resolution image is obtained by super-resolution of the second low resolution image using the currently trained target super-resolution network. The second unit is configured to calculate the patch similarity L between the first low-resolution image and the third pseudo-high-resolution image using a patch loss function. GAN The third pseudo-high resolution image is obtained by super-resolution of the first low resolution image using the currently trained target super-resolution network. The third unit is configured to calculate the total training loss generated in the first and second units using the total loss function; The judgment module determines whether the target super-resolution network in the step has converged. If it has, the training ends and the current target super-resolution network is output. If not, the target super-resolution network trained at the moment is used to construct a new pseudo-high resolution image from the first low-resolution image, and the new pseudo-high resolution image is input into the degradation network training module.

Images