Compressive sensing reconstruction method and device based on deep matrix factorization network

Abstract

The application relates to a compressed sensing reconstruction method and device based on a deep matrix factorization network. The method comprises the following steps: inputting an image to be processed based on a preset discriminant estimator, and outputting a saliency feature map; splitting the image to be processed to obtain an image block to be processed; calculating a block sampling rate corresponding to the image to be processed according to the saliency feature map and the image block to be processed; calculating a truncated sampling matrix according to the image to be processed and the block sampling rate; sampling and reconstructing the image to be processed according to the block sampling rate and the truncated sampling matrix to obtain a first reconstructed image; and optimizing the reconstruction of the first reconstructed image and the block sampling rate based on a preset deep matrix factorization reconstruction network, and outputting a second reconstructed image.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to a compressed sensing reconstruction method and device based on a deep matrix factorization network. BACKGROUND

[0002] The deep matrix factorization network can realize efficient image reconstruction, greatly reducing the sampling rate and storage requirements while maintaining high reconstruction quality. Therefore, it has wide application in many fields, including but not limited to nuclear magnetic resonance, radar image acquisition, single image machine imaging, wireless remote monitoring, etc.

[0003] In the field of signal processing, compressed sensing is an effective technology aimed at reconstructing the original image from a small number of measured signals, while reducing the need for data transmission and storage while maintaining the quality of the reconstructed image. The core idea of compressed sensing is to utilize the sparsity and low rank of the image, and to reconstruct the image through sparse representation or low rank representation, while requiring high-quality reconstruction of the image at different sampling rates.

[0004] However, the prior art requires a variety of artificially set sampling matrices, which have low matching degree with the sampling rate. The image feature extraction and sampling rate calculation algorithm cannot fully extract and utilize the image information features, and the neural network receptive field is limited and cannot fully utilize the image information, resulting in low running efficiency and accuracy of the algorithm. SUMMARY

[0005] To solve the above problems, the purpose of the present application is to provide a compressed sensing reconstruction method and device based on a deep matrix factorization network.

[0006] According to a first aspect of an embodiment of the present application, a compressed sensing reconstruction method based on a deep matrix factorization network is provided, comprising: inputting a to-be-processed image based on a preset discriminant estimator, and outputting a saliency feature map; the saliency feature map represents the information amount distribution on the to-be-processed image; splitting the to-be-processed image to obtain to-be-processed image blocks; calculating the block sampling rate corresponding to the to-be-processed image according to the saliency feature map and the to-be-processed image blocks; calculating a truncated sampling matrix according to the to-be-processed image blocks and the block sampling rate; the truncated sampling matrix is derived from a unified sampling matrix compatible with any sampling rate; sampling and reconstructing the to-be-processed image according to the block sampling rate and the truncated sampling matrix to obtain a first reconstructed image; and optimizing the reconstruction of the first reconstructed image and the block sampling rate based on a preset deep matrix factorization reconstruction network, and outputting a second reconstructed image.

[0007] Further, the preset discriminative estimator comprises two 3*3 convolutional layers and three residual blocks; wherein, the residual block comprises two 3*3 convolutional layers and a ReLU activation function.

[0008] Further, the truncated sampling matrix is calculated according to the to-be-processed image block and the block sampling rate, comprising: converting the to-be-processed image block into an image matrix; according to the image matrix, a learnable uniform sampling matrix is calculated according to Eckart-Young-Mirsky theorem and SVD decomposition; and a truncated sampling matrix is calculated according to the block sampling rate and the learnable uniform sampling matrix.

[0009] Further, the first reconstructed image and the block sampling rate are optimized and reconstructed based on the preset deep matrix factorization reconstruction network, and a second reconstructed image is output, comprising: the deep matrix factorization reconstruction network comprises a gradient descent algorithm, an iterative network, a feature extractor, a U-Net network and a learnable NAFBlock; a block gradient descent reconstructed image is determined according to the gradient descent algorithm and the first reconstructed image; an iterative image is obtained based on the iterative network according to the first reconstructed image, the block gradient descent reconstructed image and preset initial network parameters; a feature image is obtained based on the feature extractor and the U-Net network according to the iterative image and the block sampling rate; and a second reconstructed image is obtained based on the learnable NAFBlock according to the feature image.

[0010] Further, the block gradient descent reconstructed image is determined according to the gradient descent algorithm and the first reconstructed image, specifically: a gradient image block is obtained according to the following formula:

[0011]

[0012] In the formula, k represents the number of iterations, ρ k represents a learnable step size, represents a truncated sampling matrix, C i represents a random measurement value of the i-th image block. In the k-th iteration, the first reconstructed image X k-1 is divided into l non-overlapping image blocks is calculated represents a gradient image block, y i represents a calculated measurement value; the gradient image blocks are spliced to obtain a block gradient descent reconstructed image.

[0013] Further, the first reconstructed image and the block sampling rate are optimized and reconstructed based on the preset deep matrix factorization reconstruction network, and a second reconstructed image is output, comprising: a second reconstructed image is iteratively calculated according to the following formula:

[0014] U:U k+1 = (S k + ηX k )v k (λI+ η(V k ) T V k ) -1 ,

[0015] v:v k+1 = (S k + ηX k ) T U k+1 (λI+ η(U k+1 ) T U k+1 ) -1 .

[0016] S:S k+1 = S k + γ k (X k -U k+1 (V k+1 ) T )

[0017]

[0018] where λ,η denote penalty functions, S denotes a Lagrange multiplier matrix, U,V denote network parameter variables, X k denotes a first reconstructed image, γ k denotes a learnable step size, Z k+1 denotes a block gradient descent reconstructed image, k denotes an iteration number, denotes a feature extractor, the network comprises three encoders and three decoders, denotes a learnable NAFBlock, S k denotes a unit matrix, I denotes a unit matrix, D denotes a block sampling rate distribution map composed of the block sampling rate, denotes an iterative image, denotes a feature image, X k+1 denotes a second reconstructed image.

[0019] According to a second aspect of the embodiments of the present application, a compressed sensing reconstruction device based on a deep matrix factorization network is provided, and has the characteristics that the device comprises a saliency feature discrimination module configured to input a to-be-processed image based on a preset discrimination estimator and output a saliency feature map; the saliency feature map represents the information quantity distribution on the to-be-processed image; an image splitting module configured to split the to-be-processed image to obtain to-be-processed image blocks; a block sampling rate calculation module configured to calculate the block sampling rates corresponding to the to-be-processed image according to the saliency feature map and the to-be-processed image blocks; a truncated sampling matrix generation module configured to calculate a truncated sampling matrix according to the to-be-processed image blocks and the block sampling rates; the truncated sampling matrix is from a unified sampling matrix compatible with any sampling rate; a preliminary reconstruction module configured to sample and reconstruct the to-be-processed image according to the block sampling rates and the truncated sampling matrix to obtain a first reconstructed image; and an image generation module configured to optimize the reconstruction of the first reconstructed image and the block sampling rates based on a preset deep matrix factorization reconstruction network and output a second reconstructed image.

[0020] Further, the truncated sampling matrix generation module is specifically configured to: convert the to-be-processed image blocks into image matrices; calculate a learnable unified sampling matrix according to the Eckart-Young-Mirsky theorem and SVD decomposition based on the image matrices; and calculate a truncated sampling matrix according to the block sampling rates and the learnable unified sampling matrix.

[0021] According to a third aspect of the embodiments of the present application, an electronic device is provided, which comprises a processor, a memory for storing processor-executable instructions, and the processor is configured to read the executable instructions from the memory and execute the instructions to implement the compressed sensing reconstruction method based on the deep matrix factorization network provided in the first aspect of the present application.

[0022] According to a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided, which stores computer program instructions, and the program instructions are executed by a processor to implement the steps of the compressed sensing reconstruction method based on the deep matrix factorization network provided in the first aspect of the present application.

[0023] The technical solutions provided by the embodiments of the present application can have the following beneficial effects:

[0024] The application determines the information amount size distribution of the image in the to-be-processed image through a preset discriminant estimator, splits and calculates the to-be-processed image, obtains the block sampling rate corresponding to the to-be-processed image according to the saliency feature map, calculates the corresponding sampling rate according to the information amount of different image regions, reduces the sampling operation under the condition of ensuring the necessary sampling rate of the image, and improves the accuracy and running efficiency of the algorithm result of the application. The truncation sampling matrix obtained according to the block sampling rate can be compatible with any sampling rate, and adaptive sampling is achieved. The block sampling rate and the corresponding truncation sampling matrix are mutually adapted, and the to-be-processed image is sampled and reconstructed together, so that the quality of the first reconstructed image is improved, and the quality of the subsequent second reconstructed image is improved. The preset depth matrix factorization reconstruction network relies on the low-rank structure of the image, fully excavates the global information of the image, optimizes and iteratively reconstructs the image, outputs the second reconstructed image, improves the utilization rate of the global image information, and further realizes higher-quality image reconstruction. Through the preset sampling and reconstruction algorithm and network, the algorithm is improved according to the characteristics of the to-be-processed image, and a high-quality reconstructed image can be obtained with high running efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the one or more embodiments of the present specification or the prior art, the drawings needed to be used in the embodiment or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments described in the present specification, and other drawings can also be obtained by those skilled in the art without creative labor.

[0026] Figure 1 is a use flowchart of a compressed sensing reconstruction method based on a depth matrix factorization network according to an exemplary embodiment;

[0027] Figure 2 is a discriminant estimator structure schematic diagram of a compressed sensing reconstruction method based on a depth matrix factorization network according to an exemplary embodiment;

[0028] Figure 3a is a depth matrix factorization reconstruction network flowchart schematic diagram of a compressed sensing reconstruction method based on a depth matrix factorization network according to an exemplary embodiment;

[0029] Figure 3b is a gradient descent network flowchart schematic diagram of a compressed sensing reconstruction method based on a depth matrix factorization network according to an exemplary embodiment;

[0030] Figure 3c is a matrix factorization network flowchart schematic diagram of a compressed sensing reconstruction method based on a depth matrix factorization network according to an exemplary embodiment;

[0031] Figure 3d Figure 8 is a matrix factorization network parameter updating part flow diagram of a compressed sensing reconstruction method based on a deep matrix factorization network according to an example embodiment;

[0032] Figure 4 Figure 9 is a training test flow chart of a compressed sensing reconstruction method based on a deep matrix factorization network according to an example embodiment;

[0033] Figure 5 Figure 10 is a traditional comparison result chart of a compressed sensing reconstruction method based on a deep matrix factorization network according to an example embodiment;

[0034] Figure 6 Figure 11 is a deep learning comparison result chart of a compressed sensing reconstruction method based on a deep matrix factorization network according to an example embodiment;

[0035] Figure 7 Figure 12 is a model noise robustness comparison result chart of a compressed sensing reconstruction method based on a deep matrix factorization network according to an example embodiment;

[0036] Figure 8 Figure 13 is a schematic diagram of a compressed sensing reconstruction device based on a deep matrix factorization network according to an example embodiment. DETAILED DESCRIPTION

[0037] In order to make the technical scheme in one or more embodiments of the present specification better understood to those skilled in the art, the technical scheme in one or more embodiments of the present specification will be clearly and completely described below with reference to the drawings in one or more embodiments of the present specification. Obviously, the described embodiments are only a part of the embodiments of the present specification, rather than all the embodiments. Based on one or more embodiments of the present specification, all other embodiments obtained by those skilled in the art without creative labor should belong to the protection scope of the present document.

[0038] In the prior art, there are two main methods for compressive sensing methods. The first method is a compressive sensing model based on traditional methods, which mainly uses the sparsity of signals. Although these traditional methods can achieve image reconstruction tasks at various sampling rates, they require a sampling matrix that matches the sampling rate. These methods lack a certain noise robustness and are limited by the design of parameters in the algorithm, which relies on the inherent experience of researchers. The reconstructed image is prone to block artifacts, blurring, and other shortcomings. The second method is a compressive sensing model based on deep learning (which can be divided into two subcategories). The first method is an end-to-end network that quickly realizes image compressive sensing reconstruction. The second method is a content-aware reconstruction method based on a network that combines image saliency. The receptive field of the neural network is limited and cannot fully utilize image information. The near-mapping semantic segmentation network greatly increases the model parameters and the reconstruction time of the reconstructed image, resulting in low algorithm efficiency and accuracy.

[0039] In this application, a truncated sampling matrix with a matching sampling rate is automatically generated to improve the adaptability of sampling and image features, and to match subsequent high-quality reconstruction results. A preset discriminant estimator is designed to generate a corresponding saliency feature map, and a novel block sampling rate allocation strategy is developed to adapt the sampling rate to the image feature quantity. Global image information is fully utilized, a kernel norm is added for low-rank constraint, matrix factorization is used to solve the kernel norm minimization problem, a deep unfolding convolutional neural network and a U-Net network are used, and the alternating direction multiplier method is used to iteratively update the matrix factors. In the updating process, the global features of the image are fully utilized for image reconstruction to improve the running efficiency and accuracy of the algorithm.

[0040] Example Method One

[0041] As shown in Figure 1 , the embodiments of the present application provide a compressive sensing reconstruction method based on a deep matrix factorization network, comprising the following steps:

[0042] In step S101, based on a preset discriminant estimator, an image to be processed is input, and a saliency feature map is output.

[0043] The saliency feature map represents the information distribution on the image to be processed. The more information a region has, the more detailed and textured it is, and the higher the saliency is.

[0044] The discriminant estimator can make a better saliency judgment on different regions of the image by learning the details and textures of the image, and generate a saliency feature map of the corresponding image.

[0045] The preset discriminant estimator, as shown in Figure 2 , includes two 3x3 convolution layers (3x3Conv) and three residual blocks (Residual Block).

[0046] wherein the residual block comprises two 3x3 convolution layers and a ReLU activation function. The bias in the convolution layers is all set to zero.

[0047] In the residual block, the input image to be processed passes through a 3x3 convolution layer, a ReLU activation function, and another 3x3 convolution layer. Finally, the two images of the input and output are added together as the final output of the residual block.

[0048] In the embodiments of the present application, the discriminative estimator is based on a data-driven manner, i.e., a deep learning manner using a data set for adaptive learning, to realize adaptive evaluation of image detail texture and generate a saliency feature map F of the corresponding image. The saliency feature map F has the same size HxW as the image to be processed X. In the application, H=W is set.

[0049] Each pixel point of the saliency feature map F is normalized to limit the pixel value range to between 0 and 1 to match the subsequent block sampling rate allocation strategy.

[0050] Specifically, normalization is performed by a softmax operation, and the formula is as follows:

[0051]

[0052] wherein z i represents the pixel point value at the corresponding position in the image to be processed X.

[0053] In step S102, the image to be processed is split to obtain an image block to be processed.

[0054] In the embodiments of the present application, the image to be processed X is split into l image blocks x i , where l=H / BxW / B. Each image block x i has a size of BxB, and B=32 is set. BxB=N matches the subsequent block sampling rate allocation strategy.

[0055] In step S103, the block sampling rate corresponding to the image to be processed is calculated according to the saliency feature map and the image block to be processed.

[0056] Based on the block sampling rate allocation strategy, the block sampling rate corresponding to the image to be processed is calculated according to the saliency feature map and the image block to be processed. The block sampling rate allocation strategy is as follows:

[0057] According to the image block to be processed, the saliency feature map, and the random measurement value, a measurement map is calculated. According to the image block to be processed, the measurement map, the saliency feature map, and the random measurement value, an appropriate sampling rate is allocated for different saliency regions through the Adam algorithm and the simulated annealing algorithm.

[0058] Specifically, the sum-pooling operation is performed according to the saliency feature map F to obtain the assignment weight map A, the to-be-processed image block, the saliency feature map, and the random measurement value, the measurement map P is calculated, and the formula is as follows:

[0059] P = Cl x A

[0060] In the formula, C is randomly selected from (0, N), and N is the width of the sampling matrix, N = 1024.

[0061] The sampling rate of each to-be-processed image block is initialized as C / N, and therefore, the total sampling rate is Cl / Nl = C / N. Based on the idea of error iterative calibration, all elements in the measurement map P are clipped to the range of (0, N) in each iteration of the calibration process by using the Adam algorithm and the simulated annealing algorithm. The idea of iterative optimization and error correction is integrated, and the block sampling rate corresponding to the to-be-processed image is calculated The block sampling rate distribution map D is generated, in which each corresponding image block is assigned with a corresponding sampling rate according to the saliency feature map and the block sampling rate assignment algorithm. The more red the color is, the more significant the region is, and the higher the required sampling rate is. The more blue the color is, the less significant the region is, and the lower the required sampling rate is.

[0062] The block sampling rate assignment strategy effectively assigns appropriate sampling rates to different saliency regions, realizes adaptive assignment of image block sampling rates, and adapts the block sampling rate to the corresponding truncated sampling matrix. In subsequent calculation, the to-be-processed image is sampled and reconstructed together, the quality of the first reconstructed image is improved, and the quality of the subsequent second reconstructed image is improved.

[0063] In the embodiments of the present application, specifically, based on the block sampling rate assignment strategy, the process of calculating the block sampling rate corresponding to the to-be-processed image includes:

[0064] The saliency feature map F, the size of each block (Block size B), and the measurement size (Measurementsize C) are input.

[0065] Setting: l is the number of blocks, the calculation method is the product of the quotient obtained by dividing the height H and the width W of the feature map by the size B of the block: l = H / B x W / B, the learning rate decay factor of the adaptive moment estimation algorithm is a = 0.9, the first moment estimation decay factor of the adaptive moment estimation algorithm is b1 = 0.9, the second moment estimation decay factor of the adaptive moment estimation algorithm is b2 = 0.999, the initial value of the temperature parameter of the simulated annealing algorithm is t = 1, and the minimum value of the temperature parameter of the simulated annealing algorithm is t min= 1e-4, initial value of first and second moment estimation of adaptive moment estimation algorithm, dimension is same as feature map m, v = zero_matrix(F), learning rate of adaptive moment estimation algorithm lr = 1e-5, initial value of iteration number j = 0, threshold of iteration number J = 10, enter adaptive moment estimation algorithm when exceeding the threshold, initial value of measurement map P is Cl x A.

[0066] Calculate:

[0067] j = j + 1

[0068] P = round(clip 0,1024 (P))

[0069] δ = average(P) - C

[0070] In the formula, δ is the error, the difference between the average of the measurement map average(P) and the measurement value C, clip is the clipping operation, clipping the elements in the measurement map P to 0-1024, and round is the rounding operation.

[0071] If δ = 0, break the loop;

[0072] Otherwise if j <= J, P = P - δ;

[0073] Otherwise, execute the adaptive moment estimation algorithm (Adam algorithm):

[0074] grad = δ / l

[0075] m = β1 x m + (1 - β1) x grad

[0076] v = β2 x v + ((1 - β2) x grad) 2

[0077]

[0078]

[0079]

[0080] In the formula, grad is the gradient, m is the updated first moment estimation, v is the updated second moment estimation, and are the corrected first and second moment estimations, is the updated measurement distribution map.

[0081] Execute the simulated annealing algorithm to generate a polynomial distribution matrix Δ with a parameter |(δl)|;

[0082]

[0083]

[0084]

[0085] δ = δ1- δ2

[0086] wherein is the difference of the measurement map P and the product of the polynomial distribution matrix Δ and the sign function sign(δ), δ1is the mean value of the difference is the square of the difference of the measurement value size C, and δ2is the mean value of the measurement map and the value updated by the adaptive moment estimation algorithm is the square of the difference of the measurement value size C, and δ is the error.

[0087] If δ≤0, the measurement map P is directly updated as The measurement map corresponding to the smaller error is selected, so as to avoid affecting the accuracy of the reconstruction result.

[0088] Otherwise, the annealing probability of the simulated annealing algorithm is A random number ω is generated;

[0089] If ω<Y, that is, the random number is less than the annealing probability, the measurement map P is directly updated as The updated measurement map is accepted under a certain probability, so as to explore a better solution space.

[0090] t = t × α

[0091] t = min(t, t min )

[0092] D = P / B 2

[0093] wherein t = t × α is the temperature parameter of the updated simulated annealing algorithm; t = min(t, t min ) compares t with the minimum temperature t min and takes the smaller value, so as to ensure that the temperature parameter will not be too low, so that the annealing probability gradually decreases, so as to more tend to accept the solution with the worse change in the subsequent iteration.

[0094] In step S104, a truncated sampling matrix is calculated according to the to-be-processed image block and the block sampling rate.

[0095] The truncated sampling matrix is from a unified sampling matrix compatible with any sampling rate.

[0096] The to-be-processed image block is converted into an image matrix, which is convenient for subsequent calculation.

[0097] The to-be-processed image block wherein N bdenotes the number of training images, W is converted into a matrix Q with a scale size of (lN b ,B 2 ). Wherein, l denotes the number of image blocks of a training image, B represents the height of the image block, and the height and width of the image block are consistent. At this time, the length and width of Q are not equal, and Q is converted into a square matrix using the transpose idea, that is: Q T Q.

[0098] According to the image matrix, the Eckart-Young-Mirsky theorem and SVD decomposition are used to calculate the learnable uniform sampling matrix.

[0099] According to the Eckart-Young-Mirsky theorem, an optimization function formula is constructed, wherein the optimization function formula is as follows:

[0100]

[0101] In the formula, Φ is the uniform sampling matrix.

[0102] The specific solving process is as follows: according to the Eckart-Young-Mirsky theorem, the uniform sampling matrix Φ is equal to the transpose matrix U T of the left singular matrix of the SVD decomposition of Q T , and the specific calculation process is as follows:

[0103]

[0104] In the formula, V represents the right singular matrix.

[0105] The generated uniform sampling matrix has the characteristics of the image to be processed and can be compatible with any sampling rate.

[0106] According to the sampling rate distribution map composed of the block sampling rates, the learnable uniform sampling matrix Φ is truncated to obtain a truncated sampling matrix

[0107] In step S105, according to the block sampling rates and the truncated sampling matrix, the image to be processed is sampled and reconstructed to obtain a first reconstructed image.

[0108] According to the truncated sampling matrix and the image block x i , the measurement value y i of the corresponding block is calculated, and the formula is as follows:

[0109]

[0110] Through the measurement value y iand the transpose of the truncated sampling matrix, to obtain a first reconstructed image block, as shown in the following formula:

[0111]

[0112] Due to the non-full rank characteristics of the sampling matrix, the first reconstructed image block is different from the image block x i to be processed.

[0113] The first reconstructed image block is spliced to obtain a first reconstructed image X 0 .

[0114] In step S106, based on the preset depth matrix factor decomposition reconstruction network, the first reconstructed image and the block sampling rate are optimized to output a second reconstructed image.

[0115] As Figures 3a-3d shown, Figure 3a the depth matrix factor decomposition network is composed of 13 stages, each stage respectively performs block gradient descent Figure 3b and matrix factor decomposition update Figure 3c , Figure 3d operation.

[0116] The purpose of setting a multi-stage deep model is to gradually extract higher-level features through multi-layer nonlinear transformation, so as to better capture the internal structure of the reconstructed image.

[0117] According to the gradient descent algorithm and the first reconstructed image, a block gradient descent reconstructed image is determined, aiming to find the optimal solution closest to the original image.

[0118] According to the following formula, a gradient image block is obtained:

[0119]

[0120] In the formula, k represents the iteration number, ρ k represents a learnable step size, represents a truncated sampling matrix, C i represents the random measurement value of the i-th image block. In the k-th iteration, the first reconstructed image X k-1 is divided into l non-overlapping image blocks to obtain , which represents a gradient image block, y i represents the calculated measurement value.

[0121] According to the gradient descent algorithm and the first reconstructed image, a block gradient descent reconstructed image is determined; according to the first reconstructed image, the block gradient descent reconstructed image and preset initial network parameters, an iterative image is obtained based on the iterative network; according to the iterative image and the block sampling rate, a feature image is obtained based on the feature extractor and the U-Net network; and according to the feature image, a second reconstructed image is obtained based on the learnable NAFBlock.

[0122] The second reconstructed image is obtained by iterative calculation according to the following formula:

[0123] U:U k+1 =(S k +ηX k )V k (λI+η(V k ) T V k ) -1

[0124] V:V k+1 =(S k +ηX k ) T U k+1 (λI+η(U k+1 ) T U k+1 ) -1

[0125] S:S k+1 =S k +γ k (X k -U k+1 (V k+1 ) T )

[0126]

[0127] In the formula, λ and η represent penalty functions, S represents a Lagrange multiplier matrix, U and V represent network parameter variables, X k represents the first reconstructed image, γ k represents a learnable step size, Z k+1 represents a block gradient descent reconstructed image, k represents the number of iterations, represents a feature extractor, the network includes three encoders and three decoders, represents a learnable NAFBlock, S k represents a unit matrix, I represents a unit matrix, and D represents a block sampling rate distribution map composed of the block sampling rate, represents an iterative image, represents a feature image, and Xk+1 represents the second reconstructed image.

[0128] Specifically, the formula calculation process is as follows:

[0129] In order to make full use of the low-rank structure information of the image to be processed, a nuclear norm constraint approximation is applied instead of the image rank constraint. Therefore, the compressed sensing reconstruction optimization objective function can be represented as follows:

[0130]

[0131] In the formula, the return minimum index function is represented by , and λ represents the penalty function.

[0132] The nuclear norm minimization decomposition is complex, and a matrix factorization is introduced to replace the nuclear norm minimization decomposition, and the formula is as follows:

[0133]

[0134] In the formula, s.t. represents "constrained to".

[0135] A Lagrange multiplier matrix S is introduced, and the formula is as follows:

[0136]

[0137] By introducing a learnable step size γ k , and an alternating direction method of multipliers (ADMM). It can be converted into the following four update formulas:

[0138]

[0139] These variables cooperate together in the iterative update process to explore the low-rank structure of the image, and then use the global information of the image to perform high-quality reconstruction. When using the low-rank structure of the image, the required rank used can be specified (the rank used is much smaller than the length and width of the image), thereby saving model parameters and image reconstruction time.

[0140] Two backbone networks of a deep unfolded convolutional neural network (UCN) and a U-Net are used to iteratively update the four variables U, V, S, and X. First, the initial values of the variables U and V need to be obtained. In order to obtain reliable initial values of the variables U and V, it is proposed to perform singular value decomposition (SVD) on X 0 . In order to ensure the stability and robustness of the SVD of X 0 , a small perturbation ε is introduced, which is ensured to be not equal to 0, and the range of ε is between 0 and 1, and a normalization operation is used to reduce the running memory of the model. Specifically, it can be represented as:

[0141]

[0142] where Normalize is a normalization function, denotes the perturbed image, X Nor denotes the normalized image, ∑ d denotes the singular value matrix.

[0143] The initial values of U, V are: and are obtained by:

[0144]

[0145]

[0146] where and are composed of the first r columns of U and V, is a real number field, denotes an Hxr real matrix, ∑ r denotes a matrix composed of the first r rows and r columns of the singular value matrix.

[0147] In the embodiments of the present application, it is stipulated that H = W. Therefore, U and V can be updated by the following steps:

[0148]

[0149] (S 0 + ηX 0 ) V 0 and (V 0 ) T V 0 are contained in the update formula of U 1 . Therefore, they are taken as the input of the U update module and integrated. In addition, U 0 is also taken as the input of the U update module. S 0 is a unit matrix. For the U update module, these variables S 0 V 0 , X 0 V 0 , (V 0 ) T V 0 , U 0 , are updated according to the four update formulas after the introduction of the learning step size γ k and the alternating direction multiplier method. The V update module adopts a similar method. U 1 , (S 0 ) T U1 , (X 0 ) T U 1 , (U 1 ) T U 1 and V 0 , the input V update module is used for generating V 1 . Therefore, the formula of the U and V update modules can be expressed as follows:

[0150]

[0151] The update process of the variable S can be expressed as:

[0152] S: S 1 = S 0 + γ 0 (X 0 -U 1 (V 1 ) T )

[0153] Therefore, the update formula of the S update module can be expressed as:

[0154]

[0155] In the embodiments of the present application, the network structure of the UCN mainly consists of a convolutional neural network and a Mish activation function. Among them, the convolutional neural network is built for the purpose of deepening the model, aiming to obtain higher-level semantic information (higher-level texture features) of the image. The purpose of using the Mish activation function is that the Mish activation function can provide better nonlinear fitting ability and stability of the gradient.

[0156] For the update of the variable X: first, the feature extractor is used to generate a block sampling rate distribution map D' of three channels, which is used to guide the repair of the reconstructed image. In addition, a learnable NAFBlock is introduced to denoise the reconstructed image. These processes can be expressed as:

[0157]

[0158] The network structure of the feature extractor is shown in FIG. 3(c). The feature extractor consists of one convolutional layer (3x3Conv), two residual blocks (Residual Block) and another convolutional layer (3x3Conv).

[0159] The main idea of updating the variable X is to combine the feature extractor and .

[0160] It includes three encoders and three decoders. Each encoder consists of one convolutional layer, two residual blocks and one 2x2 stride convolution (SConv) layer, while each decoder includes one 2x2 transpose convolution (TConv) layer, two residual blocks and one convolutional layer. It consists of four scales. Each scale has an identity-connected skip connection between 2x2 SConv down-sampling and 2x2 TConv up-sampling operations. The number of channels of each layer from the first scale to the fourth scale is 16, 32, 64 and 128, respectively.

[0161] It is worth noting that, is unbiased, meaning that no bias is used in all Conv, SConv and TConv layers. Updating the variable in the proposed DMFNet can not only effectively capture the global structure information, but also improve the generalization ability of matrix factorization.

[0162] In summary, in the embodiment of the present application, the specific use process of the preset deep matrix factorization reconstruction network outputting the second reconstructed image is as follows:

[0163] Input the first reconstructed image X 0 And the block sampling rate distribution map D.

[0164] Let the regularization parameters γ, λ, η = 0.5, the iteration number k = 0, K = 13.

[0165] When k < K, calculate:

[0166]

[0167]

[0168]

[0169] If k = 0, calculate:

[0170]

[0171]

[0172]

[0173]

[0174]

[0175]

[0176]

[0177]

[0178] Update k <- k + 1, get the second reconstructed image X k .

[0179] where, denotes the unfolding operation, and F(·) is the folding operation.

[0180] Example method two

[0181] As Figure 4 shown, the embodiment of the present application gives a compressed sensing reconstruction training and testing method based on deep matrix factorization network, including the following steps:

[0182] In step S401, input image data and model algorithm hyperparameters.

[0183] The image data includes: training set and test set, wherein the training set is based on matlab programming software, converts the training image into mat file format for saving, and is loaded into GPU memory. The training set is composed of 25600 images of 128x128 images randomly cropped from dataset T91 and Train400. The test set is composed of datasets Set5, Set11, Set14, BSD100, and Urban100. These test sets have rich detail textures and can provide a more fair evaluation of the reconstruction performance of the model.

[0184] The model algorithm hyperparameters include: batch size, training cycle number, learning rate, momentum decay rate, model stage number, image reconstruction rank size, activation function, optimizer, data augmentation parameters, etc.

[0185] In step S402, according to the input image data and model algorithm hyperparameters, the preprocessed image data is obtained, and a unified sampling matrix is generated.

[0186] First, stretch and transpose the training set and multiply it, then perform SVD decomposition, and take the transpose matrix of the left singular matrix as the unified sampling matrix.

[0187] Standardize the natural image, scale the numerical range to 0 to 1, and appropriately use random flip, rotation, and other data enhancement operations. The natural image is the input untreated image.

[0188] According to the batch size set in advance, the data is grouped into batches. First, a batch of natural images is formed, and then a batch of natural images and reconstructed images is formed to obtain preprocessed image data, which is used as input for the model algorithm and as a target image for L2 loss function calculation to calculate the distance loss between the two.

[0189] In step S403, the pre-processed image data is image-adaptively sampled and first reconstructed.

[0190] The training image is input to the discriminative estimator to output a saliency feature map with detailed texture through adaptive learning.

[0191] The saliency feature map is subjected to block sampling rate calculation, and more sampling rate is allocated to areas with higher saliency and less sampling rate is allocated to areas with lower saliency, so as to achieve adaptive allocation of image block sampling rate.

[0192] The image block is adaptively sampled in combination with the unified sampling matrix and the block sampling rate allocation map.

[0193] The sampled image block is reconstructed based on a mathematical formula, and the specific operation is to obtain a first reconstructed image block by inverting the sampling matrix and multiplying it by the measurement signal, and to obtain a first reconstructed image by splicing the first reconstructed image block.

[0194] In step S404, the first reconstructed image is decomposed by a depth matrix factor to obtain a second reconstructed image.

[0195] The first reconstructed image is subjected to a block operation, and an image block gradient descent algorithm is used to seek an optimal solution for the reconstructed image to approximate the original image.

[0196] The image output by the image block gradient descent algorithm is subjected to matrix factor decomposition to explore the low-rank structure of the image, and then the global information of the image is fully utilized for reconstruction. A deep flattened convolutional neural network and a U-Net network are combined with the guidance of a saliency feature map and an alternating direction multiplier method to perform iterative updating to obtain a second reconstructed image.

[0197] In step S405, a loss function is calculated.

[0198] The L2 norm loss in the spatial domain (Content Loss) is calculated.

[0199] The reconstructed image of the model and the natural image are taken in pairs as the input of the L2 norm loss function in the spatial domain.

[0200] In step S406, the model is optimized.

[0201] The code is implemented based on the PyTorch deep learning framework, and the parameter gradient value in the model can be automatically calculated through backward gradient propagation starting from the final calculated loss function value.

[0202] Using the parameter gradient values, the learnable parameters in the model algorithm are updated using an optimizer (such as Adam, AdamW, SGD optimizer in PyTorch, etc.).

[0203] The above execution steps are repeated until the model reaches the training period set by the hyperparameters, and the training process of the model is stopped after the training period is reached, and the model weights are saved after each training period ends.

[0204] In step S407, the model is tested and evaluated.

[0205] The natural images of the test set are read, the natural images are used as input to test the model, loaded into the GPU memory, and the same standardization operation as in the training process is performed. Note that cropping and image enhancement are not required during testing. According to the sampling rate set during testing, the natural images are sampled and the reconstructed images are output.

[0206] The peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) commonly used in image reconstruction are used to evaluate the quality of the reconstructed images, and the reconstruction performance of the model is objectively and effectively evaluated.

[0207] If the evaluation result does not meet the expected requirements, the hyperparameters of the model, the reconstruction algorithm, etc. need to be adjusted, and the first step of the execution step is returned to retrain the model. If the evaluation result meets the requirements, the model weights can be used to obtain an image reconstruction solution.

[0208] Example method three

[0209] The present application improves the algorithm according to the characteristics of the image to be processed, and can obtain a high-quality reconstructed image with high running efficiency.

[0210] For image CS (Compressed Sensing), some traditional CS reconstruction methods, such as DWT, MH, TV, and BCS-SPL, can achieve the reconstruction task at various CS ratios, but require a corresponding sampling matrix related to the CS ratio. In addition, these traditional reconstruction algorithms rely too much on prior information. On the other hand, the parameter design of these algorithms relies too much on the inherent experience of researchers. Therefore, they are limited by high complexity and lack of adaptability.

[0211] The comparative study includes a total of 8 methods, including eight traditional CS methods, including DAMP, FOCUSS, MH, TV, BCS-SPL, MS-BCS, DWT, and GSR.

[0212] AsFigure 5 As shown, under different datasets and using different sampling rates, the PSNR (decibels) and SSIM of the proposed method (DMFNet) and the traditional compressed sensing method are compared. It can be seen that the proposed method has higher image quality.

[0213] PSNR: Peak Signal-to-Noise Ratio. The higher the PSNR value, the better the image quality.

[0214] SSIM: Structural Similarity, used to measure the similarity between two given images.

[0215] CS Ratio: Sampling rate.

[0216] Furthermore, for the evaluation of eight deep learning-based CS models, including ReconNet, ISTA-Net, ISTA-Net+, ISTA-Net++, CSNet+, COAST, AMP-Net-9BM, OPINE-Net+, and DMFNet, benchmark datasets, including Set5, Set11, Set14, BSD100, and Urban100, were extensively used. These datasets contain a variety of image features and can effectively provide results for a fair evaluation of the reconstruction performance of different models.

[0217] like Figure 6 As shown, for different sampling rates and using different datasets, the results of PSNR (dB) and SSIM are compared. It can be seen that the method in this application produces higher image quality.

[0218] Furthermore, a comparison of the model's robustness to noise is performed. For example... Figure 7 As shown, PSNR and SSIM are compared on BSD100 using different noise levels σ and different sampling rates. It can be seen that the image results obtained by the method in this application are better.

[0219] Noise robustness refers to the ability of a model to perform well on data containing some random noise without excessive fluctuations due to the presence of noise.

[0220] Exemplary device

[0221] In the embodiments of this application, such as Figure 8 It includes a salient feature discrimination module 801, an image segmentation module 802, a block sampling rate calculation module 803, a truncation sampling matrix generation module 804, a preliminary reconstruction module 805, and an image generation module 806.

[0222] The saliency feature discrimination module 801 is used to take an image to be processed as input based on a preset discrimination estimator and output a saliency feature map; the saliency feature map represents the information distribution on the image to be processed;

[0223] an image splitting module 802, configured to split the to-be-processed image to obtain to-be-processed image blocks;

[0224] a block sampling rate calculation module 803, configured to calculate a block sampling rate corresponding to the to-be-processed image according to the saliency feature map and the to-be-processed image block;

[0225] a truncated sampling matrix generation module 804, configured to calculate a truncated sampling matrix according to the to-be-processed image block and the block sampling rate; the truncated sampling matrix is from a unified sampling matrix compatible with any sampling rate; the to-be-processed image is split to obtain to-be-processed image blocks; the to-be-processed image blocks are converted into image matrices; according to the image matrices, a learnable unified sampling matrix is calculated according to Eckart-Young-Mirsky theorem and SVD decomposition; and the truncated sampling matrix is calculated according to the block sampling rate and the learnable unified sampling matrix.

[0226] a preliminary reconstruction module 805, configured to sample and reconstruct the to-be-processed image according to the block sampling rate and the truncated sampling matrix to obtain a first reconstructed image;

[0227] an image generation module 806, configured to optimize reconstruction of the first reconstructed image and the block sampling rate based on a preset deep matrix factorization reconstruction network, and output a second reconstructed image.

[0228] An exemplary electronic device

[0229] The embodiment provides an electronic device, which includes one or more processors, and internal memory and external memory, the internal memory stores instructions, when the instructions are executed by the one or more processors, the one or more processors execute the compression sensing reconstruction method based on the deep matrix factorization network as described in any of the preceding embodiments.

[0230] The processor is configured to execute all or part of the steps of the compression sensing reconstruction method based on the deep matrix factorization network as described in the embodiments. The memory is configured to store various types of data, which may, for example, include instructions of any application program or method in the electronic device, and application program related data.

[0231] The processor can be an Application Specific Integrated Cricuit (ASIC), a Digital Signal Processor (DSP), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a controller, a microcontroller, a microprocessor, or other electronic components implementing the described embodiments of the deep matrix factorization network based compressive sensing reconstruction.

[0232] Computer storage media

[0233] A computer program is stored on the computer readable storage medium, and when executed by one or more processors, implements the deep matrix factorization network based compressive sensing reconstruction as described in any of the preceding embodiments.

[0234] The above description has described certain embodiments of the present document. Other embodiments are within the scope and spirit of the claims. In some instances, the acts or steps can be performed in different order than described and still achieve desirable results. Additionally, the process depicted in the figures can not require the particular order shown or sequential order to achieve desirable results. In certain implementations, multitasking and parallel processing can be advantageous.

[0235] The above description is only a summary of the embodiments of the present document and is not used to limit the present document. For those skilled in the art, the present document can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present document shall be included in the scope of the claims of the present document.

Claims

1. A compressed sensing reconstruction method based on a deep matrix factorization network, characterized in that, include: Based on a preset discriminant estimator, the input image to be processed outputs a salient feature map; The saliency feature map represents the information distribution on the image to be processed; The image to be processed is split into blocks to be processed; Based on the saliency feature map and the image block to be processed, calculate the block sampling rate corresponding to the image block to be processed; Based on the image block to be processed and the block sampling rate, a truncated sampling matrix is ​​calculated; the truncated sampling matrix comes from a unified sampling matrix compatible with any sampling rate. Based on the sampling rate of each block and the truncated sampling matrix, the image to be processed is sampled and reconstructed to obtain a first reconstructed image; Based on a preset depth matrix factorization reconstruction network, the first reconstructed image and the block sampling rate are optimized and reconstructed to output a second reconstructed image; The reconstruction network based on a preset depth matrix factorization optimizes the reconstruction of the first reconstructed image and the block sampling rate, and outputs a second reconstructed image, including: The deep matrix factorization reconstruction network includes a gradient descent algorithm, an iterative network, a feature extractor, a U-Net network, and a learnable NAFBlock; Based on the gradient descent algorithm and the first reconstructed image, a block gradient descent reconstructed image is determined; Based on the first reconstructed image, the block gradient descent reconstructed image, and preset initial network parameters, an iterative image is obtained based on the iterative network. Based on the iterative image and the block sampling rate, a feature image is obtained based on the feature extractor and the U-Net network. Based on the feature image and the learnable NAFBlock, a second reconstructed image is obtained.

2. The method according to claim 1, characterized in that, The preset discrimination estimator includes: The preset discriminant estimator includes two 3×3 convolutional layers and three residual blocks; The residual block comprises two 3×3 convolutional layers and a ReLU activation function.

3. The method according to claim 1, characterized in that, The step of calculating the truncated sampling matrix based on the image block to be processed and the block sampling rate includes: Convert the image block to be processed into an image matrix; Based on the image matrix, a learnable uniform sampling matrix is ​​calculated according to the Eckart-Young-Mirsky theorem and SVD decomposition. The truncated sampling matrix is ​​calculated based on the block sampling rate and the learnable uniform sampling matrix.

4. The method according to claim 3, characterized in that, The step of determining the block gradient descent reconstructed image based on the gradient descent algorithm and the first reconstructed image specifically involves: The gradient image patch is obtained using the following formula: In the formula, Indicates the number of iterations. Indicates the learnable step size. This indicates the truncated sampling matrix. Indicates the first Random measurements of image patches, at the th In the next iteration, the first reconstructed image Classified as Non-overlapping image blocks Calculations yielded Represents gradient image patches, This indicates the calculated measurement value; The gradient image blocks are stitched together to obtain a block gradient descent reconstructed image.

5. The method according to claim 1, characterized in that, The reconstruction network based on a preset depth matrix factorization optimizes the reconstruction of the first reconstructed image and the block sampling rate, and outputs a second reconstructed image, including: In the formula, Represents the penalty function, Represents the Lagrange multiplier matrix. Represents network parameter variables. This represents the first reconstructed image. Indicates the learnable step size. This represents a block gradient descent image reconstruction. Indicates the number of iterations. Indicates feature extractor, The U-Net network consists of three encoders and three decoders. This indicates a learnable NAFBlock. Describe an identity matrix. Represents the identity matrix. This represents a block sampling rate distribution map composed of the block sampling rates. Represents an iterative image. Representing feature images, This represents the second reconstructed image.

6. A compressed sensing reconstruction device based on a deep matrix factorization network, characterized in that, include: The saliency feature discrimination module is used to take the image to be processed as input and output a saliency feature map based on a preset discrimination estimator. The saliency feature map represents the information distribution on the image to be processed; The image segmentation module is used to segment the image to be processed to obtain image blocks to be processed; The block sampling rate calculation module is used to calculate the block sampling rate corresponding to the image to be processed based on the saliency feature map and the image block to be processed; The truncated sampling matrix generation module is used to calculate a truncated sampling matrix based on the image block to be processed and the block sampling rate; the truncated sampling matrix comes from a unified sampling matrix compatible with any sampling rate; The preliminary reconstruction module is used to sample and reconstruct the image to be processed according to the sampling rate of each block and the truncated sampling matrix to obtain a first reconstructed image; The image generation module is used to optimize the reconstruction of the first reconstructed image and the block sampling rate based on the preset depth matrix factorization reconstruction network, and output the second reconstructed image. The reconstruction network based on a preset depth matrix factorization optimizes the reconstruction of the first reconstructed image and the block sampling rate, and outputs a second reconstructed image, including: The deep matrix factorization reconstruction network includes a gradient descent algorithm, an iterative network, a feature extractor, a U-Net network, and a learnable NAFBlock; Based on the gradient descent algorithm and the first reconstructed image, a block gradient descent reconstructed image is determined; Based on the first reconstructed image, the block gradient descent reconstructed image, and preset initial network parameters, an iterative image is obtained based on the iterative network. Based on the iterative image and the block sampling rate, a feature image is obtained based on the feature extractor and the U-Net network. Based on the feature image and the learnable NAFBlock, a second reconstructed image is obtained.

7. The apparatus according to claim 6, characterized in that, The truncated sampling matrix generation module is specifically used for: The image block to be processed is converted into an image matrix; based on the image matrix, a learnable uniform sampling matrix is ​​calculated according to the Eckart-Young-Mirsky theorem and SVD decomposition; and a truncated sampling matrix is ​​calculated based on the block sampling rate and the learnable uniform sampling matrix.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the compressed sensing reconstruction method based on a deep matrix factorization network as described in any one of claims 1 to 5 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the compressed sensing reconstruction method based on a deep matrix factorization network as claimed in any one of claims 1 to 5.

Images