Image reconstruction method and system based on deep learning

Abstract

The embodiment of the invention relates to the technical field of image processing, and discloses an image reconstruction method based on deep learning, and the method comprises the steps: compressing an original image f; reconstructing the compressed original image f, and determining a target function according to the Bayesian theorem; training a neural network model according to the objective function until a loss index of the neural network model meets a threshold condition; and reconstructing the image according to the trained neural network model. Compared with a traditional method, the method provided by the invention can provide higher compression ratio and better image reconstruction quality by using a deep learning technology.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image reconstruction method and system based on deep learning. Background Art

[0002] With the rapid development of information technology, data acquisition and processing have become one of the key technologies in many fields. Traditional data acquisition methods follow the Nyquist sampling theorem, requiring the sampling frequency to be at least twice the highest frequency of the signal to ensure that the signal can be reconstructed without distortion; however, this method has the problems of inefficiency and high cost when processing high-dimensional data.

[0003] In order to solve this problem, the compressed sensing theory was proposed in 2006, providing a new perspective for data acquisition and signal recovery. The core idea of ​​the compressed sensing theory is to project high-dimensional data into a low-dimensional space for acquisition, and then reconstruct the original signal from these low-dimensional data through a specific algorithm. This method is based on a basic assumption: many natural signals can be compressed on a certain sparse basis, that is, most of the signal's energy is concentrated in a few non-zero coefficients.

[0004] In the field of imaging, high-resolution detectors are usually very expensive, and the computational burden for collecting and processing large amounts of data is often very heavy; therefore, compressed sensing theory and its single-pixel camera technology have shown great advantages in this field. By reducing the use of high-resolution cameras, system costs are reduced, and at the same time, image quality is guaranteed through efficient compression acquisition and image reconstruction algorithms. In this process, the selection and optimization of image reconstruction methods are particularly important, which is directly related to the quality and efficiency of the final imaging. Summary of the invention

[0005] Based on this, it is necessary to propose an image reconstruction method based on deep learning to address the above problems.

[0006] A deep learning-based image reconstruction method, the method comprising:

[0007] Compress the original image f;

[0008] Reconstructing the compressed original image f and determining the objective function according to the Bayesian theorem;

[0009] Training the neural network model according to the objective function until the loss index of the neural network model meets the threshold condition;

[0010] The image is reconstructed based on the trained neural network model.

[0011] In the above solution, compressing the original image f specifically includes: generating a compression model according to the measurement signal g, the encoding matrix Φ and the original image f:

[0012] p(Φ; f; g) = p(Φ)p(f)p(g)

[0013] Where p is the probability distribution function.

[0014] In the above scheme, after obtaining the compression model, the likelihood estimate of the compression model is obtained:

[0015]

[0016] in,‖·‖ 2 , λ is a parameter, i is the i-th image block, d is the number of image blocks, g i is the i-th measurement signal, and Φ is the encoding matrix.

[0017] In the above solution, after obtaining the likelihood estimate of the compression model, the measurement signal g is decomposed:

[0018]

[0019] Among them, D KL is the KL divergence between the variational approximation q(f, Φ|g) and the true posterior distribution p(f, Φ|g), E q(f，Φ|g) is the expectation of the variational approximation distribution q(f, Φ|g).

[0020] In the above scheme, the compressed original image f is reconstructed, and the objective function is determined according to the Bayesian theorem, specifically including:

[0021] The number of preset solutions, assuming the reconstructed image With a prior distribution:

[0022]

[0023] Among them, N(·;μ,σ 2 ) has a mean of μ and a variance of σ 2 Gaussian distribution of , I is the unit matrix, i is the i-th training image;

[0024] Construct a variational form of the reconstructed image posterior and bring it close to the posterior distribution

[0025] Assume that the reconstructed image And the encoding matrix Φ is independently distributed:

[0026]

[0027] Estimate the posterior distribution in terms of parameters Determine the reconstructed image The variational posterior form of :

[0028]

[0029] Among them, W R is the parameter of the deep Bayesian network R, g is the network input measurement signal, μ i (g; W R ) and m 2 (g,W R ) are respectively the mean and variance of the reconstructed image;

[0030] Get the objective function:

[0031]

[0032] in, is the divergence loss function, is the average error loss function, Is a constant.

[0033] In the above scheme, the variational form of constructing the reconstructed image posterior and bring it close to the posterior distribution Also includes:

[0034] The first item This constitutes a variational lower bound on the boundary likelihood p(g):

[0035]

[0036] in,

[0037] In the above scheme, the first likelihood term is:

[0038]

[0039]

[0040] Among them, const is a constant and λ is a parameter;

[0041] The second KL divergence term is parsed as:

[0042]

[0043] Among them, σ is the variance, m i is the mean of the reconstructed image.

[0044] In the above scheme, μ is 0, σ 2 1×10-5 , i is the i-th training image, i=1,…d.

[0045] In the above scheme, the encoding matrix Φ satisfies the loss function: Where d is the number of image blocks, I is the unit matrix, Φ T is the transpose of the encoding matrix.

[0046] The present application also proposes an image reconstruction system based on deep learning, the system comprising: an objective function acquisition unit, a model training unit and an image reconstruction unit;

[0047] The objective function acquisition unit is used to compress the original image f; reconstruct the compressed original image f, and determine the objective function according to the Bayesian theorem;

[0048] The model training unit is used to train the neural network model according to the objective function until the loss index of the neural network model meets the threshold condition;

[0049] The image reconstruction unit is used to reconstruct the image according to the trained neural network model.

[0050] The embodiment of the present invention has the following beneficial effects: first compressing the original image f; reconstructing the compressed original image f, and determining the objective function according to Bayes' theorem; training the neural network model according to the objective function until the loss index of the neural network model meets the threshold condition; reconstructing the image according to the trained neural network model; this method uses a deep learning model and data advantages to improve the image reconstruction quality, and can provide a higher compression rate and better image reconstruction quality than traditional methods, which is conducive to improving image reconstruction efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] in:

[0053] Figure 1 is a flowchart of an image reconstruction method based on deep learning in one embodiment;

[0054] Figure 2 Schematic diagram of target image compression measurement process;

[0055] Figure 3Schematic diagram of deep Bayesian network;

[0056] Figure 4 Based on Figure 1 Reconstruction renderings. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0058] In the following description, a large number of specific details are given in order to provide a more thorough understanding of the present invention; however, it is obvious to those skilled in the art that the present invention can be implemented without one or more of these details; in other examples, in order to avoid confusion with the present invention, some technical features known in the art are not described, and it should be understood that the present invention can be implemented in different forms and should not be interpreted as limited to the embodiments presented here; on the contrary, providing these embodiments will make the disclosure thorough and complete and fully convey the scope of the present invention to those skilled in the art.

[0059] The purpose of the terms used herein is only to describe specific embodiments and is not intended to be limiting of the present invention. When used herein, the singular forms "a", "an" and "said / the" are also intended to include the plural forms, unless the context clearly indicates otherwise, and it should also be understood that the terms "consisting of" and / or "comprising", when used in this specification, determine the presence of the features, integers, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or groups. When used herein, the term "and / or" includes any and all combinations of the relevant listed items.

[0060] In recent years, deep learning has developed rapidly in academia and industry. It has achieved significant improvements in recognition rates in many traditional signal representation and recognition tasks, demonstrating its ability to handle complex recognition tasks. It has attracted a large number of scholars to study its theory and applications, and many fields have begun to try to use deep learning to solve some problems in their fields.

[0061] In deep learning, convolutional neural networks, stacked denoising autoencoders and other networks have excellent signal feature representation capabilities and can be used in the field of compressed sensing. Through a large number of training samples, the structural characteristics of real signals can be accurately learned, not limited to the sparsity of the signal, to improve the accuracy of signal reconstruction. On the other hand, with the support of parallel GPU computing hardware conditions, the operation time of deep learning networks in high dimensions is guaranteed, and the compressed sensing reconstruction process can be converted into deep neural network calculations to achieve real-time reconstruction. In the field of deep learning, convolutional neural networks have powerful problem-solving capabilities, so almost all image-related research uses a large number of convolutional layers as the basic module for solving different types of problems, and autoencoders have excellent performance in high-dimensional and low-dimensional data processing.

[0062] To facilitate understanding, the relevant terms involved in this application are first introduced below.

[0063] (1) Deep Variational Bayes refers to a technique that combines deep learning and variational Bayesian inference. In deep learning, we often use large neural network models to handle complex tasks, but these models are prone to overfitting on limited data sets and the generalization ability of the models may be weak. To solve this problem, Bayesian Deep Learning was proposed, which attempts to introduce model uncertainty into neural networks to improve the robustness and generalization ability of the models.

[0064] (2) Prior distribution. In Bayesian statistics and Bayesian deep learning, the prior distribution represents our belief or assumption about a parameter or a set of parameters before seeing any data. The prior distribution is the probability distribution of the possible values ​​of the parameter, which reflects our "prior" knowledge of the possible values ​​of these parameters;

[0065] (3) The posterior distribution, which describes the probability distribution of the unknown parameters given the observed data. The posterior distribution is the result of Bayesian inference and is calculated by combining the prior distribution and the likelihood function.

[0066] (4) Encoding matrix. In machine learning and data science, the encoding matrix can be viewed as a matrix with K rows and L columns consisting of the codes of all categories, where K represents the number of sample categories in the multi-classification task and L represents the length of the code. This means that the multi-classification task is divided into L binary classification tasks. Generally, "codeword" is used to refer to the codes in the encoding matrix. During the training process, each binary classification task divides the data set according to the positive and negative class labels of the corresponding columns of the encoding matrix and trains the classifier, eventually generating L binary classifiers.

[0067] In order to thoroughly understand the present invention, a detailed structure will be presented in the following description to illustrate the technical solution proposed by the present invention; optional embodiments of the present invention are described in detail as follows, but in addition to these detailed descriptions, the present invention may also have other implementation methods.

[0068] like Figure 1 As shown, in one embodiment, a deep learning-based image reconstruction method is provided, and the deep learning-based image reconstruction method includes steps S101 to S104, which are described in detail as follows:

[0069] S101, compressing the original image f;

[0070] In some embodiments, compressing the original image f specifically includes:

[0071] Generate a compression model based on the measurement signal g, the encoding matrix Φ and the original image f:

[0072] p(Φ; f; g) = p(Φ)p(f)p(g)

[0073] Where p is the probability distribution function.

[0074] Specifically, the above compression model is generated under the Bayesian framework.

[0075] Preferably, after obtaining the compression model, obtain a likelihood estimate of the compression model:

[0076]

[0077] in,‖·‖ 2 , λ is a parameter, i is the i-th image block, d is the number of image blocks, g i is the i-th measurement signal, and Φ is the encoding matrix.

[0078] In some embodiments, after obtaining the likelihood estimate of the compression model, the measurement signal g is decomposed:

[0079]

[0080] Among them, D KL is the KL divergence between the variational approximation q(f, Φ|g) and the true posterior distribution p(f, Φ|g), E q(f，Φ|g) is the expectation of the variational approximation distribution q(f, Φ|g).

[0081] like Figure 2 Shown is: the above-mentioned target image compression measurement process.

[0082] S102, reconstructing the compressed original image f, and determining the target function according to the Bayesian theorem;

[0083] The objective function is the key to the optimization problem. It defines how to evaluate the performance of the neural network model. In the image reconstruction task, the objective function may involve pixel-level errors, structural similarity, or other perceptual quality indicators. After determining the objective function, it can ensure that the neural network model is optimized in the right direction during the training process, thereby producing high-quality reconstructed images.

[0084] In some embodiments, the compressed original image f is reconstructed, and the objective function is determined according to the Bayesian theorem, specifically including:

[0085] (1) The number of preset solutions, assuming the reconstructed image With a prior distribution:

[0086]

[0087] Among them, N(·;μ,σ 2 ) has a mean of μ and a variance of σ 2 Gaussian distribution of , I is the unit matrix, i is the i-th training image;

[0088] Among them, this step is to reconstruct the image It should be as close to the original image f as possible, so the number of solutions is limited, assuming With a prior distribution, specifically, μ is 0, σ 2 1×10 -5 , i is the i-th training image, i=1,…d.

[0089] (2) Constructing a variational form of the reconstructed image posterior and bring it close to the posterior distribution

[0090] (3) Assume that the reconstructed image And the encoding matrix Φ is independently distributed:

[0091]

[0092] (4) Estimate the posterior distribution in the form of parameters Determine the reconstructed image The variational posterior form of :

[0093]

[0094] Among them, W R is the parameter of the deep Bayesian network R, g is the network input measurement signal, μ i (g; W R ) and m 2 (g,W R ) are the mean and variance of the reconstructed image respectively;

[0095] (5) Obtain the objective function:

[0096]

[0097] in, is the divergence loss function, is the average error loss function, Is a constant.

[0098] 10. Preferably, the encoding matrix Φ satisfies the loss function: Where d is the number of image blocks, I is the unit matrix, Φ T is the transpose of the encoding matrix.

[0099] In some embodiments, a variational form of the reconstructed image posterior is constructed and bring it close to the posterior distribution Also includes:

[0100] The first item This constitutes a variational lower bound on the boundary likelihood p(g):

[0101]

[0102] in,

[0103] Preferably, the first likelihood term is:

[0104]

[0105] Among them, const is a constant and λ is a parameter;

[0106] The second KL divergence term is parsed as:

[0107]

[0108] Among them, σ is the variance and mi is the mean of the reconstructed image.

[0109] S103, training the neural network model according to the objective function until the loss index of the neural network model meets the threshold condition;

[0110] Training a neural network model is an iterative process in which the model is adjusted based on the performance of the objective function on the training data. In the image reconstruction task, the training data may include the original image and its corresponding compressed version. By training the neural network model, it can learn the complex mapping required to recover the original image from the compressed data, generalize to unseen data, and produce high-quality reconstructed images.

[0111] like Figure 3As shown in the figure, the neural network model is a deep Bayesian network, which includes a sampling subnetwork, an initial reconstruction subnetwork and a deep reconstruction subnetwork. The sampling subnetwork consists of a 3×3 convolutional layer, and the training weight is the optimized coding aperture. The initial reconstruction subnetwork also consists of a 3×3 convolutional layer to realize the upsampling operation from two-dimensional measurement values ​​to three-dimensional data cubes. The deep reconstruction subnetwork consists of a 3×3 convolutional layer, a feature space information supplementation module and a 1×1 convolutional layer to effectively remove noise during the reconstruction process and obtain high-quality reconstructed images.

[0112] Preferably, the above threshold conditions can be manually set and adjusted according to different determining factors such as data and usage scenarios.

[0113] S104: Reconstruct the image according to the trained neural network model.

[0114] Once the neural network model is trained and meets the performance threshold, it can be used for actual image reconstruction tasks, which include receiving new compressed images as input and outputting the corresponding reconstructed images. Since the neural network model has been trained to optimize a specific objective function, it should be able to produce high-quality, realistic reconstructed images. In addition, due to the powerful capabilities of deep learning models, this method may be able to handle a variety of different types of images and compression algorithms.

[0115] like Figure 4As shown: the images labeled 0 and 1 are original images, the images labeled a and g are images processed by the real-time learning network, and the corresponding PSNR / SSIM of this group of images are 28.09 / 0.8864 and 2939 / 0.8759 respectively; the images labeled b and k are images processed by the interpretable deep network inspired by optimization, and the corresponding PSNR / SSIM of this group of images are 29.34 / 0.9155 and 31.17 / 0.9121 respectively; the images labeled c and l are images processed by combining average pooling and max pooling. The images processed by the neural network architecture with two pooling strategies, the corresponding PSNR / SSIM of this group of images are 29.19 / 0.9054 and 31.28 / 0.9096 respectively; the images labeled d and m are images processed by the COAST neural network, and the corresponding PSNR / SSIM of this group of images are 28.30 / 0.8929 and 30.04 / 0.8848 respectively; the images labeled e and n are images processed by the neural network architecture based on the classic U-Net architecture, but with some dynamic and guided elements added, and the corresponding PSNR / SSIM of this group of images are 29.38 / 0.9214 and 31.92 / 0.9189 respectively; the images labeled f and o are images processed by the hybrid architecture model based on Transformer, and the corresponding PSNR / SSIM of this group of images are 29.43 / 0.9134 and 31.98 / 0.9184 respectively; the images labeled g and p are based on the reference article: Dynamic The images processed by Path-Controllable Deep Unfolding Network for Compressive Sensing have a PSNR / SSIM of 29.05 / 0.9040 and 30.88 / 0.8977 respectively; the images labeled h and q are images processed based on the reference article: Deep memory-augmented proximal unrolling network for compressive sensing, and the PSNR / SSIM of this group of images are 28.98 / 0.9032 and 30.97 / 0.8990 respectively; the images labeled i and r are reconstruction effect diagrams of the present invention, and the PSNR / SSIM of this group of images are 29.78 / 0.9257 and 32.20 / 0.9204 respectively; it can be seen that compared with other deep learning reconstruction algorithms, the PSNR and SSIM of the reconstruction results of the present invention are both optimal, and it can also be seen from the local enlarged image of the reconstruction result that the detailed texture reconstruction of the present invention is better restored, without block effect and blur.

[0116] In summary, the image reconstruction method based on deep learning proposed in this application establishes the probability distribution of the reconstructed image, then obtains the maximum likelihood estimate of the compressed observation model, and then constructs the variational form of the posterior distribution to approximate the posterior distribution of the reconstructed image, and then designs a deep reconstruction network to learn the parameters of the variational form. Finally, the deep network loss function is set to the variational lower bound of the boundary likelihood, so as to obtain the reconstructed target image through network training.

[0117] The present application also proposes an image reconstruction system based on deep learning, which includes: an objective function acquisition unit, a model training unit and an image reconstruction unit;

[0118] The target function acquisition unit is used to compress the original image f; reconstruct the compressed original image f, and determine the target function according to the Bayesian theorem;

[0119] A model training unit, used to train the neural network model according to the objective function until the loss index of the neural network model meets the threshold condition;

[0120] The image reconstruction unit is used to reconstruct the image according to the trained neural network model.

[0121] Those of ordinary skill in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory.

[0122] Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0123] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0124] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent of the present application. It should be pointed out that, for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. The above disclosure is only a preferred embodiment of the present invention, and of course it cannot be used to limit the scope of the rights of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope covered by the present invention.

Claims

1. A deep learning-based image reconstruction method, characterized in that: The method comprises: Compress the original image f; Reconstructing the compressed original image f and determining the objective function according to the Bayesian theorem; Training the neural network model according to the objective function until the loss index of the neural network model meets the threshold condition; The image is reconstructed based on the trained neural network model.

2. The image reconstruction method based on deep learning according to claim 1, characterized in that: The compressing the original image f specifically includes: Generate a compression model based on the measurement signal g, the encoding matrix Φ and the original image f: p(Φ; f; g) = p(Φ)p(f)p(g) Where p is the probability distribution function.

3. The image reconstruction method based on deep learning according to claim 2, characterized in that: After obtaining the compression model, obtain a likelihood estimate of the compression model: Among them, ||·|| 2 , λ is a parameter, i is the i-th image block, d is the number of image blocks, g i is the i-th measurement signal, and Φ is the encoding matrix.

4. The image reconstruction method based on deep learning according to claim 2, characterized in that: After obtaining the likelihood estimate of the compression model, the measurement signal g is decomposed: log p(g,Φ,f)=∫q(f,Φ|g)log p(g|f,Φ)df =E q(f，φ，g) [log p(g|f,Φ)p(f)-log q(f,Φ|g)]+D KL (q(f,Φ|g)||p(f,Φ|g)) =l(g,Φ,f)+D KL (q(f,Φ|g)|||p(f,Φ|g)) Among them, D KL is the KL divergence between the variational approximation q(f, Φ|g) and the true posterior distribution p(f, Φ|g), E q(f，Φ|g) is the expectation of the variational approximation distribution q(f, Φ|g).

5. The image reconstruction method based on deep learning according to claim 3, characterized in that: The reconstructing the compressed original image f and determining the objective function according to the Bayesian theorem specifically includes: The number of preset solutions, assuming the reconstructed image With a prior distribution: Among them, N(·;μ,σ 2 ) has a mean of μ and a variance of σ 2 Gaussian distribution of , I is the unit matrix, i is the i-th training image; Construct a variational form of the reconstructed image posterior and bring it close to the posterior distribution Assume that the reconstructed image And the encoding matrix Φ is independently distributed: Estimate the posterior distribution in terms of parameters Determine the reconstructed image The variational posterior form of : Among them, W R is the parameter of the deep Bayesian network R, g is the network input measurement signal, μ i (g; W R ) and m 2 (g; W R ) are respectively the mean and variance of the reconstructed image; Get the objective function: Among them, l KL is the divergence loss function, l MSE is the average error loss function, Is a constant.

6. The image reconstruction method based on deep learning according to claim 5, characterized in that: The variational form of constructing the reconstructed image posterior and bring it close to the posterior distribution Also includes: The first item This constitutes a variational lower bound on the boundary likelihood p(g): in, 7. The image reconstruction method based on deep learning according to claim 6, characterized in that: The first likelihood term is: Among them, const is a constant and λ is a parameter; The second KL divergence term is parsed as: Among them, σ is the variance, m i is the mean of the reconstructed image.

8. The image reconstruction method based on deep learning according to claim 5, characterized in that: The μ is 0, σ 2 1×10 -5 , i is the i-th training image, i=1,…d.

9. The image reconstruction method based on deep learning according to claim 8, characterized in that: The encoding matrix Φ satisfies the loss function: Where d is the number of image blocks, I is the unit matrix, Φ T is the transpose of the encoding matrix.

10. An image reconstruction system based on deep learning, characterized in that: The system comprises: an objective function acquisition unit, a model training unit and an image reconstruction unit; The objective function acquisition unit is used to compress the original image f; reconstruct the compressed original image f, and determine the objective function according to the Bayesian theorem; The model training unit is used to train the neural network model according to the objective function until the loss index of the neural network model meets the threshold condition; The image reconstruction unit is used to reconstruct the image according to the trained neural network model.

Images