Image compression sensing method and device based on multi-stage information transmission enhancement

Abstract

The application discloses an image compression sensing method and device based on multi-stage information transmission enhancement. Firstly, an image is divided into blocks, then the image blocks are decomposed into low-frequency subbands and high-frequency subbands by using a discrete wavelet transform, and are pulled into vectors; next, sampling is performed according to the characteristics of different subbands, a linear inverse problem is solved by using an approximate message passing algorithm, then a reconstruction model is established by combining the prior of a wavelet tree structure and the prior of multi-stage information, and the model is expanded into a neural network; therefore, the transmission and compensation capabilities of information can be improved, the quality of current recovery is effectively improved by extracting valuable features from each stage in the past, the representation capability of the network is greatly improved, and the information loss in different stages is minimized.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more particularly to an image compression sensing method based on multi-stage information transmission enhancement, and an image compression sensing device based on multi-stage information transmission enhancement. Background Technology

[0002] Compressed sensing is a signal processing technique that allows signals to be sampled and reconstructed at rates far below the Nyquist sampling rate. The basic idea behind compressed sensing is to utilize the sparsity of signals to acquire and reconstruct sparse or compressible signals. By leveraging this sparsity, compressed sensing can perform accurate signal reconstruction using nonlinear algorithms even at extremely low sampling rates. Compressed sensing reduces the amount of information that needs to be transmitted, significantly reducing bandwidth limitations in special environments, and is widely used in fields such as medical imaging, wireless communication, and remote sensing.

[0003] Despite significant progress in image compression sensing based on deep learning, and the recent deep unrolling models integrating the benefits of optimization algorithms and deep networks to maintain accuracy, interpretability, and speed, deep unrolling methods suffer from several problems: (1) Existing deep unrolling models often rely solely on the reconstruction results of the final stage, neglecting the multi-stage correlations between different stages, which may lead to getting stuck in a local optimum. (2) Deep unrolling models generally use convolution operations, resulting in information loss during inter-stage information transfer. This makes it easy for information from earlier stages to be forgotten, and long-term dependency issues are rarely addressed. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, the technical problem to be solved by the present invention is to provide an image compression sensing method based on multi-stage information transmission enhancement, which can improve the transmission and compensation capabilities of information, effectively improve the quality of current recovery by extracting valuable features from previous stages, greatly improve the representation capability of the network, and minimize information loss at different stages.

[0005] The technical solution of this invention is: this image compression sensing method based on multi-stage information transmission enhancement includes the following steps:

[0006] (1) Image sampling: First, the image is divided into blocks, and then the discrete wavelet transform is used to decompose the image blocks into low-frequency sub-bands and high-frequency sub-bands, and then stretched into vectors; next, sampling is performed according to the characteristics of different sub-bands:

[0007] y s，l，i =A l *θ s，l，i (1)

[0008] Where s∈{LL, HL, LH, HH}, 1≤l≤L, L is the wavelet decomposition level, y is the observation value of compressed sensing, A is the sampling matrix, and * denotes matrix multiplication;

[0009] (2) Image reconstruction based on multi-stage information: The linear inverse problem is solved by an approximate message passing algorithm, and then a reconstruction model is established by combining the prior of wavelet tree structure and the prior of multi-stage information, and then expanded into a neural network; the iterative solution of the reconstruction model is as follows:

[0010]

[0011]

[0012] Where G represents the grouping operation, and ||Gθ||2+||θ||1 is the wavelet tree structure prior. It is a multi-stage information prior.

[0013] This invention first divides the image into blocks, then uses discrete wavelet transform to decompose the image blocks into low-frequency and high-frequency sub-bands, and stretches them into vectors. Next, sampling is performed according to the characteristics of different sub-bands, and the linear inverse problem is solved through an approximate message passing algorithm. Then, a reconstruction model is established by combining the prior information of the wavelet tree structure and the prior information of multi-stage information, and unfolded into a neural network. Therefore, it can improve the information transmission and compensation capabilities, effectively improve the quality of the current recovery by extracting valuable features from previous stages, greatly improve the representation capability of the network, and minimize the information loss at different stages.

[0014] A multi-stage information transmission-enhanced image compression sensing device is also provided, the device comprising:

[0015] The image sampling module first divides the image into blocks, then uses discrete wavelet transform to decompose the image blocks into low-frequency and high-frequency sub-bands, and stretches them into vectors; next, it samples according to the characteristics of different sub-bands:

[0016] y s，l，i =A l *θ s，l，i (1)

[0017] Where s∈{LL,HL,LH,HH},1≤l≤L,L is the wavelet decomposition level, y is the observation value of compressed sensing, A is the sampling matrix, and * denotes matrix multiplication;

[0018] The image reconstruction module is based on multi-stage information: it solves the linear inverse problem using an approximate message-passing algorithm, then combines prior information from wavelet tree structure and multi-stage information to build a reconstruction model, which is then expanded into a neural network; the iterative solution of the reconstruction model is as follows:

[0019]

[0020]

[0021] Where G represents the grouping operation, and ||Gθ||2+||θ||1 is the wavelet tree structure prior. It is a multi-stage information prior. Attached Figure Description

[0022] Figure 1 A flowchart of the image compression sensing method based on multi-stage information transmission enhancement according to the present invention is shown.

[0023] Figure 2 The structure diagram of the k-th reconstruction stage is shown.

[0024] Figure 3 The Long Memory Mapped Module (CLMM) module across stages is shown.

[0025] Figure 4 The PSNR curves of the reconstruction results are shown when the number of iterations k takes different values. Detailed Implementation

[0026] like Figure 1 As shown, this image compressed sensing method based on multi-stage information transmission enhancement includes the following steps:

[0027] (1) Image sampling: First, the image is divided into blocks, and then the discrete wavelet transform is used to decompose the image blocks into low-frequency sub-bands and high-frequency sub-bands, and then stretched into vectors; next, sampling is performed according to the characteristics of different sub-bands:

[0028] y s，l，i =A l *θ s，l，i (1)

[0029] Where s∈{LL,HL,LH,HH},1≤l≤L,L is the wavelet decomposition level, y is the observation value of compressed sensing, A is the sampling matrix, and * denotes matrix multiplication;

[0030] (2) Image reconstruction based on multi-stage information: The linear inverse problem is solved by an approximate message passing algorithm, and then a reconstruction model is established by combining the prior of wavelet tree structure and the prior of multi-stage information, and then expanded into a neural network; the iterative solution of the reconstruction model is as follows:

[0031]

[0032]

[0033] Where G represents the grouping operation, and ||Gθ||2+||θ||1 is the wavelet tree structure prior. It is a multi-stage information prior.

[0034] This invention first divides the image into blocks, then uses discrete wavelet transform to decompose the image blocks into low-frequency and high-frequency sub-bands, and stretches them into vectors. Next, sampling is performed according to the characteristics of different sub-bands, and the linear inverse problem is solved through an approximate message passing algorithm. Then, a reconstruction model is established by combining the prior information of the wavelet tree structure and the prior information of multi-stage information, and unfolded into a neural network. Therefore, it can improve the information transmission and compensation capabilities, effectively improve the quality of the current recovery by extracting valuable features from previous stages, greatly improve the representation capability of the network, and minimize the information loss at different stages.

[0035] Preferably, in step (1), the number of channels in most convolutional layers of the network is set to channel = 32. The number of rows in the sampling matrix A differs for models trained at different sampling rates. Furthermore, for the initial reconstruction module, the transpose of the sampling matrix A is used... T The initial reconstruction of x is obtained from the observed value y. 0 =A T *y. In the reconstruction module, each iteration is obtained by solving the mapping through one iteration of the approximate message passing algorithm.

[0036] Preferably, in step (2), LSTM is used for long-term and short-term information modeling:

[0037]

[0038] Where * denotes a convolution operation. W represents the Hadama product. si W hi ,…,W ho b represents the filter weights. i b f b c b o Indicates the bias term, i k f k o k These represent the input gate, forget gate, and output gate, respectively. k h represents an accumulator for state information. k Output c from the latest unit k and output gate o k control.

[0039] Preferably, in step (2), to address information loss during inter-stage transmission, a cross-stage long-term memory mapping module (CLMM) is used. The CLMM consists of three parts: two convolutional units (ConvBlock) and one LSTM unit, and the set of outputs from the first k stages [x...]. 0 x1 , ..., x k As input to the first convolutional unit:

[0040]

[0041] Wherein, [·] represents feature concatenation. The ConvBlock unit is composed of a Conv-ReLU-Conv structure. This unit is used to fuse and extract useful features from the output sequences of the first k stages in order to perform high-capacity information transmission across stages.

[0042] Deep features combining information from the first k stages Input into the LSTM unit:

[0043]

[0044] [h k c k Update the current state of deep features and predict multi-stage information, capture the multi-stage correlation between different stage reconstructions, pass deep features at the same location, and achieve long-term information compensation across stages.

[0045] Preferably, in step (2), at the end of a reconstruction stage, the obtained features are further integrated and refined using the following units to improve the reconstruction quality, wherein... This is the output of the k-th stage:

[0046]

[0047] Preferably, this method uses 400 natural images with an average resolution of 481*321 from the BSDS500 image set as the training dataset. During training, 128*128 blocks are randomly cropped from each image to form a batch of 32 blocks, and the model is optimized using MSE as a quality metric. When the model is trained and optimized, it is trained for 200 epochs on an NVIDIA RTX 3090 GPU using Adam in a PyTorch 1.7, Python 3.7 environment. The learning rate is initialized to 0.0001, and the loss function L = L MSE +0.001*L texture +0.01*L init Optimize.

[0048] Preferably, during the testing phase, the same strategy as during training is used to divide the image into 128*128 image blocks for sampling and reconstruction.

[0049] Preferably, the method is evaluated on the Set11 and Urban100 test sets.

[0050] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium. When executed, the program includes the steps of the methods of the above embodiments. The storage medium can be ROM / RAM, magnetic disk, optical disk, memory card, etc. Therefore, corresponding to the method of the present invention, the present invention also includes an image compression sensing device based on multi-stage information transmission enhancement. This device is typically represented in the form of functional modules corresponding to the steps of the method. The device includes:

[0051] The image sampling module first divides the image into blocks, then uses discrete wavelet transform to decompose the image blocks into low-frequency and high-frequency sub-bands, and stretches them into vectors; next, it samples according to the characteristics of different sub-bands:

[0052] y s,l,i =A l *θ s,l,i (1)

[0053] Where s∈{LL,HL,LH,HH}, 1≤l≤L, L is the wavelet decomposition level, y is the observation value of compressed sensing, A is the sampling matrix, and * denotes matrix multiplication;

[0054] The image reconstruction module is based on multi-stage information: it solves the linear inverse problem using an approximate message-passing algorithm, then combines prior information from wavelet tree structure and multi-stage information to build a reconstruction model, which is then expanded into a neural network; the iterative solution of the reconstruction model is as follows:

[0055]

[0056]

[0057] Where G represents the grouping operation, and ||Gθ||2+||θ||1 is the wavelet tree structure prior. It is a multi-stage information prior.

[0058] The beneficial technical effects of the present invention are demonstrated through the following verification.

[0059] The method disclosed in this invention was tested and evaluated on multiple public datasets, including Set11 and Urban100. To verify the performance of the proposed image compressed sensing method based on multi-stage information transmission enhancement, the method MTADUN disclosed in this invention was compared with some advanced image compressed sensing models, including ISTA-Net++, AMP-Net, DPA-Net, OPINE-Net, COAST, DPUNet, etc. The comparison results are shown in Tables 1 and 2.

[0060] Table 1. Comparison results of average PSNR (dB) / SSIM on Set11

[0061]

[0062] Table 2. Comparison of Average PSNR (dB) / SSIM on Urban100

[0063]

[0064]

[0065] As shown in Tables 1 and 2, "Ours" represents the proposed multi-stage information transmission enhancement method. The best results are in bold, and the second-best results are underlined. Compared with other state-of-the-art methods, "Ours" achieves higher PSNR / SSIM at all sampling rates. For example, "Ours" achieves a PSNR gain of approximately 0.95–4.44 dB on the Set11 test set at 50% sampling rate, and a SSIM gain of approximately 0.0010–0.0141. Test results on the Urban100 dataset at all sampling rates also outperform other state-of-the-art methods. In summary, the method disclosed in this invention significantly improves the compressed sensing performance of images.

[0066] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention shall still fall within the protection scope of the present invention.

Claims

1. An image compression sensing method based on multi-stage information transmission enhancement, characterized in that: The method includes the following steps: (1) Image sampling: First, the image is divided into blocks, and then the discrete wavelet transform is used to decompose the image blocks into low-frequency sub-bands and high-frequency sub-bands, and then stretched into vectors; next, sampling is performed according to the characteristics of different sub-bands: y s，l，i =A l *i s，l，i (1) Where s∈{LL, HL, LH, HH}, 1≤l≤L, L is the wavelet decomposition level, y is the observation value of compressed sensing, A is the sampling matrix, and * denotes matrix multiplication; (2) Image reconstruction based on multi-stage information: The linear inverse problem is solved by an approximate message passing algorithm, and then a reconstruction model is established by combining the prior of wavelet tree structure and the prior of multi-stage information, and then expanded into a neural network; the iterative solution of the reconstruction model is as follows: Where G represents the grouping operation, and ||Gθ||2+||θ||1 is the wavelet tree structure prior. It is a multi-stage information prior.

2. The image compression sensing method based on multi-stage information transmission enhancement according to claim 1, characterized in that: In step (1), the number of channels in most convolutional layers of the network is set to channel=32. For models trained at different sampling rates, the number of rows in sampling matrix A is different. For the initial reconstruction module, the transpose A of the sampling matrix is ​​used. T The initial reconstruction of x is obtained from the observed value y. 0 =A T *y; In the reconstruction module, each iteration is obtained by solving the mapping through one iteration of the approximate message passing algorithm.

3. The image compression sensing method based on multi-stage information transmission enhancement according to claim 2, characterized in that: In step (2), LSTM is used for long-term and short-term information modeling: Where * denotes a convolution operation. W represents the Hadama product. si W hi , ..., W ho b represents the filter weights. i b f b c b o Indicates the bias term, i k f k o k These represent the input gate, forget gate, and output gate, respectively. k h represents an accumulator for state information. k Output c from the latest unit k and output gate o k control.

4. The image compression sensing method based on multi-stage information transmission enhancement according to claim 3, characterized in that: In step (2), to address information loss during inter-stage transmission, a cross-stage long-term memory mapping module (CLMM) is used. The CLMM consists of three parts: two convolutional units (ConvBlock) and one LSTM unit, and the set of outputs from the first k stages [x...]. 0 x 1 , ..., x k As input to the first convolutional unit: Wherein, [·] represents feature concatenation. The ConvBlock unit is composed of a Conv-ReLU-Conv structure. This unit is used to fuse and extract useful features from the output sequences of the first k stages in order to perform high-capacity information transmission across stages. Deep features combining information from the first k stages Input into the LSTM unit: [h k c k Update the current state of deep features and predict multi-stage information, capture the multi-stage correlation between different stage reconstructions, pass deep features at the same location, and achieve long-term information compensation across stages.

5. The image compression sensing method based on multi-stage information transmission enhancement according to claim 4, characterized in that: In step (2), at the end of a reconstruction phase, the obtained features are further integrated and refined using the following units to improve the reconstruction quality, wherein... This is the output of the k-th stage:

6. The image compression sensing method based on multi-stage information transmission enhancement according to claim 5, characterized in that: This method uses 400 natural images with an average resolution of 481*321 from the BSDS500 image set as the training dataset. During training, 128*128 blocks are randomly cropped from each image to form a batch of 32 blocks, and the model is optimized using MSE as the quality metric. When the model is trained and optimized, it is trained for 200 epochs on an NVIDIA RTX 3090 GPU using Adam in a PyTorch 1.7, Python 3.7 environment. The learning rate is initialized to 0.0001, and the loss function L = L MSE +0.001*L texture +0.01*L init Optimize.

7. The image compression sensing method based on multi-stage information transmission enhancement according to claim 6, characterized in that: During the testing phase, the same strategy as during training was used to divide the image into 128*128 image blocks for sampling and reconstruction.

8. The image compression sensing method based on multi-stage information transmission enhancement according to claim 7, characterized in that: The method was evaluated on the Set11 and Urban100 test sets.

9. An image compression sensing device based on multi-stage information transmission enhancement, characterized in that: The device includes: The image sampling module first divides the image into blocks, then uses discrete wavelet transform to decompose the image blocks into low-frequency and high-frequency sub-bands, and stretches them into vectors; next, it samples according to the characteristics of different sub-bands: y s，l,i =A l *i s，l，i (1) Where s∈{LL, HL, LH, HH}, 1≤l≤L, L is the wavelet decomposition level, y is the observation value of compressed sensing, A is the sampling matrix, and * denotes matrix multiplication; The image reconstruction module is based on multi-stage information: it solves the linear inverse problem using an approximate message-passing algorithm, then combines prior information from wavelet tree structure and multi-stage information to build a reconstruction model, which is then expanded into a neural network; the iterative solution of the reconstruction model is as follows: Where G represents the grouping operation, and ||Gθ||2+||θ||1 is the wavelet tree structure prior. It is a multi-stage information prior.

Images