Image compression sensing method and system based on wavelet transform and convolution filter flow

Abstract

The present invention provides an image compression sensing method and system based on wavelet transform and convolution filter flow. The image compression sensing method includes: obtaining multiple image block measurement values; performing initial reconstruction on the multiple image block measurement values ​​to obtain an initial reconstructed image; inputting the initial reconstructed image into a deep reconstruction network for processing to obtain a final reconstructed image, and the deep reconstruction network includes a discrete wavelet transform network, a pixel-level filter network based on a convolution filter flow, and an inverse discrete wavelet transform fusion network connected in sequence. Applying a discrete wavelet transform network to the initial reconstructed image can expand the receptive field and reduce blocking artifacts; the discrete wavelet transform network decomposes the initial reconstructed image, and uses a pixel-level filter network to perform convolution filter flows on different sub-band images, thereby realizing simultaneous analysis of high-frequency and low-frequency components, extracting image features from different frequency domain levels, and considering the local neighborhood information of image pixels, effectively improving the quality of compressed sensing image reconstruction.

Description

Technical Field

[0001] The present invention belongs to the technical field of image compression sensing, and in particular to an image compression sensing method and system based on wavelet transform and convolution filter flow. Background Art

[0002] In the study of image compressed sensing reconstruction algorithms, some traditional algorithms such as MH (such as the MH algorithm published in the paper "Image compressed sensing using convolutional neural network[J]" published by W. Shi, F. Jiang, S. Liu, etc. in IEEE Transactions on Image Processing in 2020) mostly start from image priors and add additional optimization methods or combine iterative threshold algorithms. Most of them have high computational complexity and poor image reconstruction quality. Deep learning-based algorithms such as ReconNet (such as the ReconNet algorithm published in the paper "ReconNet: Non-iterative reconstruction of images from compressively sensed measurements[C]" published by K. Kulkarni, S. Lohit, P. Turaga, etc. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) in 2016) have low computational costs, but most of them use small receptive fields, resulting in the loss of non-local information of the image itself and low reconstruction quality.

[0003] Generally speaking, there are three ways to expand the receptive field, namely increasing the network depth, increasing the filter size and using pooling operations. The first two methods will inevitably lead to higher computational costs. The difference is that the pooling operation directly reduces the spatial resolution of the feature map to ensure efficiency, but this operation will lead to information loss. To solve this problem, the dilated convolution was proposed. While keeping the input unchanged, it skips some pixels in the existing pixels and inserts "zero" in the convolution kernel to increase the spatial range seen by the convolution, thereby achieving a balance between the size of the receptive field and the computational efficiency. However, the dilated convolution does not use all pixels for calculation. Its convolution kernel lacks continuity and will lose the continuity of information, resulting in a grid effect and poor image reconstruction quality. At the same time, in the compressed sensing image reconstruction model, the high-frequency part of the image is often ignored in terms of its effect on the reconstruction effect.

[0004] In order to solve the above-mentioned problem of image information loss due to a small receptive field, the present invention aims to provide an image compression sensing method that can expand the receptive field while avoiding increasing the computational burden and sacrificing the potential information of the image, so as to improve the image reconstruction quality. Summary of the invention

[0005] The present invention aims to at least solve the technical problems existing in the prior art and provide an image compression sensing method and system based on wavelet transform and convolution filter flow.

[0006] In order to achieve the above-mentioned purpose of the present invention, according to the first aspect of the present invention, the present invention provides an image compression sensing method based on wavelet transform and convolution filter flow, including: obtaining multiple image block measurement values; performing initial reconstruction on the multiple image block measurement values ​​to obtain an initial reconstructed image; inputting the initial reconstructed image into a deep reconstruction network for processing to obtain a final reconstructed image, wherein the deep reconstruction network includes a discrete wavelet transform network, a pixel-level filter network based on a convolution filter flow, and an inverse discrete wavelet transform fusion network connected in sequence.

[0007] In order to achieve the above-mentioned purpose of the present invention, according to the second aspect of the present invention, the present invention provides an image compression sensing system based on wavelet transform and convolution filter flow, including: a sampling module, used to compress and sample the original image to obtain multiple image block measurement values; an initial reconstruction module, used to perform initial reconstruction on the multiple image block measurement values ​​to obtain an initial reconstructed image; a deep reconstruction network, used to further reconstruct the initial reconstructed image to obtain a final reconstructed image, the deep reconstruction network includes a discrete wavelet transform network, a pixel-level filter network based on the convolution filter flow and an inverse discrete wavelet transform fusion network connected in sequence.

[0008] The above technical solution: In the deep reconstruction network, the discrete wavelet transform network is applied to the initial reconstructed image, which can expand the receptive field, minimize the blocking artifacts in the sampling process of obtaining the image block measurement values, and improve the reconstruction quality; the discrete wavelet transform network decomposes the initial reconstructed image to obtain four sub-band images, and uses the pixel-level filter network to perform convolution filtering flows on different sub-band images, and learns pixel-level filters in the frequency domain, realizing simultaneous analysis of high-frequency and low-frequency components, extracting image features from different frequency domain levels, combining the wavelet domain with the pixel domain, and considering the local neighborhood information of image pixels, effectively improving the quality of compressed sensing image reconstruction, and having good interpretability and visual effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is a flowchart of an image compression sensing method in a preferred embodiment of the present invention;

[0010] Figure 2It is a detailed flow chart of an image compression sensing method in a preferred embodiment of the present invention;

[0011] Figure 3 is a structural block diagram of an image compression sensing system in a preferred embodiment of the present invention;

[0012] Figure 4 This is a schematic diagram of the structure of a sampling module in a preferred embodiment of the present invention;

[0013] Figure 5 is a schematic diagram of a pixel shuffling process in a preferred embodiment of the present invention;

[0014] Figure 6 is a schematic diagram of the structure of an initial reconstruction module in a preferred embodiment of the present invention;

[0015] Figure 7 This is a schematic diagram of a deep reconstruction network structure in a preferred embodiment of the present invention;

[0016] Figure 8 It is a schematic diagram of the network structure of the approximate low-frequency component convolution filter flow in a preferred embodiment of the present invention;

[0017] Fig. 9 It is a schematic diagram of the specific structure of the approximate low-frequency component convolution filter flow network in a preferred embodiment of the present invention;

[0018] Fig.10 It is a filter flow chart corresponding to the reconstructed image of each wavelet component in the image compression sensing system at a sampling rate of 20% in a preferred embodiment of the present invention. Description of the drawings:

[0020] C convolution layer; B batch normalization layer (BN); R activation function layer; U upsampling layer; M maximum pooling layer. DETAILED DESCRIPTION

[0021] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0022] In the description of the present invention, it is necessary to understand that the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0023] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0024] In the theory of image compression sensing, most modern techniques for studying image inverse problems focus on regularization techniques for computational optimization, effectively utilizing various image priors to obtain better image reconstruction effects, but this results in expensive computational costs. At the same time, the compressed sensing image algorithm based on deep learning effectively utilizes data-driven methods to make image reconstruction faster, but it lacks the guarantee of interpretability and robustness. Inspired by the idea of ​​convolutional filter flow in the prior art, such as the filter flow idea disclosed in the paper "Filter flow" published by SM Seitz, S. Baker et al. at the 2009 IEEE 12th International Conference on Computer Vision. Kyoto in 2009, this application proposes a new image compression sensing convolutional filter flow framework. The convolutional filter flow (CFF) not only retains good interpretability and control over the reconstruction results, but also ensures fast computational reasoning. It provides a new solution for such low-level image processing tasks in compressed sensing.

[0025] In the convolution filter flow, the pixels in the local neighborhood of the input image are linearly combined to reconstruct the pixels at the corresponding position in the output image. The difference from ordinary convolution is that the weight of the filter varies with the spatial position. This application estimates the weight of the regression filter for a given input image. In short, the convolution filter flow does not focus on the flow between the original image and the undersampled image, but focuses on the entire convolution filter flow and the transformation of the image. Although the convolution filter flow is based on a convolutional neural network, it is different from other compressed sensing reconstruction algorithms based on deep learning, because the convolution filter flow can show an explicit representation from the input image to the output reconstructed image, and how it is transformed is intuitive, transparent and explainable.

[0026] Specifically, according to the filtering operation I'(u,v) represents an image, u = (u,v) is the position of the pixel, and on each pixel of the image, a filter can be defined Acts on the image region centered at pixel u:

[0027]

[0028] Among them, i,j control the size of the neighborhood. The convolution filter flow transforms the image I' 1 →I' 2 Modeled as a linear mapping, as described above, each output pixel depends only on the local neighborhood of the corresponding input pixel. Such a flow can be found by learning methods and can be solved as a constrained linear system by the convolutional filter flow framework.

[0029] I' 2 =TI' 1 ,T∈Γ (2)

[0030] Where T is a vector matrix, each row of which defines I' 1 The linear combination of pixels in I' 2 Generate pixels at appropriate positions. T∈Γ is a placeholder for the entire set of additional constraints. For example, T corresponding to ordinary convolution is usually a circulant matrix whose rows are circulant permutations of a single filter weight set. These filter weights are usually constrained to have compact local non-zeros. Formula (2) seems simplified, but it is not well-considered to solve it directly in theory. For example, when the image size is different, the filter needs to be changed in space, which requires a learnable method.

[0031] In order to better estimate the transformation between the input image and the output image, the convolution filter stream trains a learnable function f by learning. w (·), w is the hyperparameter of the convolutional filter flow network, and the convolutional filter flow model can be expressed as:

[0032]

[0033] Need to give an image pair i=1,...,N'. In the theory of compressed sensing, the degraded image is taken as I' 1 , the original image is I' 2 , forming an image training pair to find the optimal parameter w. In the design of the loss function, we strive to minimize the reconstructed image With the original image I' 2 The difference between , the loss function is set to:

[0034]

[0035] By design w The structure of (·) is used as a hard constraint to select an appropriate regularization term. As a soft constraint. Each pixel in depends only on I' 1 The local neighborhood centered on the corresponding pixel in is equivalent to introducing a structural constraint that each pixel of the output depends only on the local neighborhood of the corresponding input pixel. The size of this neighborhood is also a hyperparameter.

[0036] The structural constraint of the local neighborhood of the input pixel is implemented through the "im2col" operation, which actually optimizes the convolution operation by converting the convolution operation between matrices into the multiplication operation between matrices. This is based on the similarity between the convolution multiplication and addition operation and the positive operation within the vector, which reduces the time and space costs and solves the problem of discontinuous memory reading caused by following the mathematical steps of the convolution operation.

[0037] Convolutional filter flow is a completely self-supervised learning framework. As an "ideal state" of deep learning, the convolutional filter flow model does not require manually labeled data and can be learned directly from unlabeled data. Given an original image block, it is compressed and sampled, and initially reconstructed to obtain an initial reconstructed image block, which forms a training pair with the original image block, and the model parameters are trained in a self-supervised manner.

[0038] Example 1

[0039] This embodiment discloses an image compression sensing method based on wavelet transform and convolution filter flow, preferably, as Figure 1 As shown, including:

[0040] Step S1, obtaining measurement values ​​of multiple image blocks;

[0041] Step S2, performing initial reconstruction on the measurement values ​​of the multiple image blocks to obtain an initial reconstructed image;

[0042] Step S3, input the initial reconstructed image into the deep reconstruction network to obtain the final reconstructed image, such as Figure 2 As shown, the deep reconstruction network includes a discrete wavelet transform network, a pixel-level filter network based on a convolutional filter flow, and an inverse discrete wavelet transform fusion network connected in sequence.

[0043] In this embodiment, in order to reduce the overall computational complexity of discrete wavelet transform and convolution filter flow network, the block-based compressive sensing (BCS) algorithm is used to perform block operations on the original image. In order to improve the sampling adaptability, Figure 4 As shown, step S1 specifically includes: inputting the original image into the sampling network to obtain multiple image block measurement values, the sampling network includes a sampling convolution layer, and the sampling convolution layer includes n s The size of the sampling convolution kernel is S×S×k. Block compressed sensing uses a size of n s × 2 The measurement matrix is ​​denoted by A s , for image block I i For example, the compressed sampling process can be expressed as i =A s I i The specific sampling process of this application is to transform the measurement matrix A s Each row in is regarded as a convolution kernel, and an unbiased convolution layer is used to simulate the block compression sampling process. Then, when the sampling rate is MN, each convolution kernel of size S×S×k is used, because the image block I i The size is also S×S×k, so the measurement matrix A s There are n s Row, where n s =MkS 2 / N, that is, n can be measured from an image block s The compressed sampling process can be expressed as:

[0044] y=S(I)=W s *I, where S(·) represents the compressed sampling operation, y represents the compressed sensing measurement value, and W s is the convolution kernel, whose size is S×S×k and has a total of n s , * represents the convolution operation, I is the input image to be measured, and the sampling network can be expressed as Figure 4 Training the measurement matrix through learning can more effectively utilize the characteristics of the image itself, such as structural information, and lay a good foundation for the subsequent reconstruction process.

[0045] In this embodiment, in order to convert the image block observation value into an image with the same size as the original image for subsequent further reconstruction, preferably, as Figure 6 As shown, step S2 specifically includes:

[0046] Input multiple image block measurement values ​​into the initial reconstruction convolution layer to obtain the vector representation of the image block; Figure 5 As shown, the vector representations of multiple image blocks are reshaped into an initial reconstructed image with the same size as the original image through a pixel shuffling operation.

[0047] In this embodiment, in the initial reconstruction network of the present application, the input is the measured value y after the compressed sampling of the output of the sampling network. Similar to the sampling network, a convolution layer is first used to extract the information of the measured value and transform it into image information. The specific method is as follows: First, A s T Set the convolution kernel of a convolution layer to a size of 1×1×n s , and there are kS 2 filters, because the measurement value y is also 1×1×n s , each image block contains kS 2 elements, the output is 1×1×kS 2 The output is a tensor of size , which is the vector representation of each initial reconstructed image block. Then, as Figure 5 As shown, the pixel shuffle operation converts 1×1×kS 2 The tensor output is reshaped into S×S×k, with a total of n s image blocks, ensuring that the initial reconstructed image is consistent with the original image size, and can achieve tensor splicing and shaping.

[0048] In this embodiment, in order to independently and simultaneously analyze high-frequency and low-frequency components, image features are extracted from different frequency domain levels, and combined with pixel-level filters to implement pixel-level regional processing to improve reconstruction quality. Further preferably, the deep reconstruction network performs:

[0049] Step S31, the initial reconstructed image is subjected to discrete wavelet transform to obtain an approximate low-frequency component subband image, a horizontal component subband image, a vertical component subband image, and a diagonal component subband image. Preferably, the discrete wavelet transform preferably but not limited to adopting a first-level Haar wavelet decomposition can minimize the computational cost of wavelet transform and inverse transform in the decomposition and synthesis process. In addition, Haar wavelet has low computational complexity and spatial complexity, introduces low edge distortion and more redundancy minimization, and due to its reversibility, this downsampling method will not lose any image information or image intermediate features, and can also capture the frequency and position information of the feature map, which helps to retain detailed texture information when using multi-frequency feature representation. The first-level Haar wavelet transform converts the initial reconstructed image from the spatial domain to the frequency domain, and extracts low and high frequency components, namely: low-frequency approximation components (Approximation components, A) subband image, horizontal components (Horizontal components, H) subband image, vertical components (Vertical components, V) subband image, diagonal components (Diagonal components, D) subband image. Then for the initial reconstructed image of size M×N M represents the width of the image, N represents the length of the image, u = (u, v) is the position of the pixel, and the wavelet decomposition equation for the low-frequency approximate component A subband image is:

[0050]

[0051] The wavelet decomposition equation for the sub-band image of horizontal component H, vertical component V, and diagonal component D is:

[0052]

[0053] Where p = {H, V, D}, The size of the four decomposed sub-band images is M / 2×N / 2. This is because the discrete wavelet transform has four mathematical operations. ψ H (u,v),ψ V (u,v),ψ D (u,v) are the wavelet functions of the low-frequency approximate component, horizontal component, vertical component and diagonal component, respectively, which are defined as:

[0054]

[0055]

[0056] It is obvious that the four wavelet functions are orthogonal to each other and form a reversible matrix of size 4×4.

[0057] Step S32: The approximate low-frequency component sub-band image, the horizontal component sub-band image, the vertical component sub-band image and the diagonal component sub-band image are reconstructed into an approximate low-frequency component sub-band reconstructed image, a horizontal component sub-band reconstructed image, a vertical component sub-band reconstructed image and a diagonal component sub-band reconstructed image respectively by using a pixel-level filter network based on a convolution filter flow. Figure 7 As shown, each component sub-band image is set with a corresponding pixel-level filter to complete the image reconstruction of the component sub-band, realizing the synchronous processing of four components. Figure 7 As shown, the pixel-level filter network includes: an approximate low-frequency component branch, including an approximate low-frequency component convolution filter flow network, a first skipping path and a first multiplication module, the approximate low-frequency component sub-band image is respectively input into the approximate low-frequency component convolution filter flow network and the first skipping path, the output end of the approximate low-frequency component convolution filter flow network and the output end of the first skipping path are both connected to the input end of the first multiplication module, and the first multiplication module outputs the approximate low-frequency component sub-band reconstructed image; a horizontal component branch, including a horizontal component convolution filter flow network, a second skipping path and a second multiplication module, the horizontal component sub-band image is respectively input into the horizontal component convolution filter flow network and the second skipping path, the output end of the horizontal component convolution filter flow network and the output end of the second skipping path are both connected to the input end of the second multiplication module, and the second multiplication module Output the horizontal component sub-band reconstructed image; the vertical component branch includes a vertical component convolution filter flow network, a third skipping path and a third multiplication module, the vertical component sub-band images are respectively input into the vertical component convolution filter flow network and the third skipping path, the output end of the vertical component convolution filter flow network and the output end of the third skipping path are both connected to the input end of the third multiplication module, and the third multiplication module outputs the vertical component sub-band reconstructed image; the diagonal component branch includes a diagonal component convolution filter flow network, a fourth skipping path and a fourth multiplication module, the diagonal component sub-band images are respectively input into the diagonal component convolution filter flow network and the fourth skipping path, the output end of the diagonal component convolution filter flow network and the output end of the fourth skipping path are both connected to the input end of the fourth multiplication module, and the fourth multiplication module outputs the diagonal component sub-band reconstructed image.

[0058] Step S33, respectively perform inverse discrete wavelet transform processing on the approximate low-frequency component sub-band reconstructed image, the horizontal component sub-band reconstructed image, the vertical component sub-band reconstructed image and the diagonal component sub-band reconstructed image, and fuse the images processed by inverse discrete wavelet transform to obtain the final reconstructed image, and the fusion method is preferably but not limited to adding four inverse discrete wavelet transform processed images.

[0059] In this embodiment, it is further preferred that after obtaining the reconstructed images of the four sub-bands, the four sub-band images are transformed back to the spatial domain using the biorthogonal characteristics of the inverse discrete wavelet transform, and the images processed by the inverse discrete wavelet transform are fused to obtain the final reconstructed image. for:

[0060]

[0061] Where (u, v) represents the pixel position; M×N represents the size of the reconstructed image and the original image; represents the approximate low-frequency component subband reconstructed image, Represents the wavelet function of the approximate low-frequency component; component index p = {H, V, D}, when index p = H, represents the horizontal component subband reconstructed image, ψ H (u,v) represents the wavelet function of the horizontal component; when index p = V, represents the vertical component subband reconstructed image, ψ(u,v) represents the wavelet function of the vertical component; when index p = D, represents the diagonal component subband reconstructed image, ψ D (u,v) represents the wavelet function of the diagonal component. The final reconstructed image The size of is M×N, which is consistent with the original image.

[0062] Example 2

[0063] This embodiment discloses an image compression sensing system based on wavelet transform and convolution filter flow. In a preferred implementation, Figure 2 As shown, including:

[0064] A sampling module is used to compress and sample the original image to obtain multiple image block measurement values; preferably, the sampling module includes a sampling convolution layer, such as Figure 4 shown.

[0065] An initial reconstruction module is used to perform initial reconstruction on the measurement values ​​of multiple image blocks to obtain an initial reconstructed image; preferably, the initial reconstruction module includes an initial reconstruction convolution layer and a pixel shuffling unit connected in sequence, such as Figure 6 As shown, the initial reconstruction process can be expressed as: in, is the initial reconstructed image, R 0 (·) represents the initial reconstruction operation, PixelShuffle(·) represents the pixel shuffle operation, and W initial kS 2 1×1×n s The size of the convolution kernel.

[0066] The deep reconstruction network is used to further reconstruct the initial reconstructed image to obtain the final reconstructed image. The deep reconstruction network includes a discrete wavelet transform network, a pixel-level filter network based on a convolution filter flow, and an inverse discrete wavelet transform fusion network connected in sequence.

[0067] In this embodiment, preferably, Figure 7As shown, the discrete wavelet transform network performs discrete wavelet transform on the initial reconstructed image to obtain an approximate low-frequency component subband image, a horizontal component subband image, a vertical component subband image and a diagonal component subband image. The discrete wavelet transform network preferably but not limited to selects a first-level Haar wavelet transform to convert the initial reconstructed image from the spatial domain to the frequency domain, and extracts low and high frequency components, namely: a low-frequency approximation component (Approximation components, A) subband image, a horizontal component (Horizontal components, H) subband image, a vertical component (Verticalcomponents, V) subband image, and a diagonal component (Diagonal components, D) subband image. The specific process has been described in detail in the previous embodiment and will not be repeated here.

[0068] In this embodiment, in order to analyze high-frequency and low-frequency components at the same time, extract image features from different frequency domain levels, and each sub-band contains different levels of information. The processing method for each sub-band should not use weight sharing technology to train all sub-bands, but should be "adapted to local conditions". Each sub-band component uses a different convolution filter flow network to obtain a filter flow graph, such as Figure 7 As shown, the pixel-level filter network includes:

[0069] The approximate low-frequency component branch includes an approximate low-frequency component convolution filter flow network, a first skipping path, and a first multiplication module. The approximate low-frequency component subband image is input into the approximate low-frequency component convolution filter flow network and the first skipping path respectively. The output end of the approximate low-frequency component convolution filter flow network and the output end of the first skipping path are both connected to the input end of the first multiplication module. The first multiplication module outputs the approximate low-frequency component subband reconstructed image. The specific reconstruction process of the approximate low-frequency component subband reconstructed image can be expressed as:

[0070] First, the convolutional filter network R approximates the low-frequency components. A (·) Approximate low-frequency component subband image of the input Processing to obtain the approximate low-frequency component filter flow graph

[0071] Finally, the first jump path is used to introduce the approximate low-frequency component subband image into the first multiplication module and the approximate low-frequency component filter flow graph to perform multiplication operations of different dimensions of the broadcast mechanism to obtain the approximate low-frequency component subband reconstructed image.

[0072]

[0073] Horizontal component branches, such as Figure 7As shown, the horizontal component branch includes a horizontal component convolution filter flow network, a second skip path and a second multiplication module. The horizontal component sub-band image is respectively input into the horizontal component convolution filter flow network and the second skip path. The output end of the horizontal component convolution filter flow network and the output end of the second skip path are both connected to the input end of the second multiplication module. The second multiplication module outputs the horizontal component sub-band reconstructed image.

[0074] Vertical component branches, such as Figure 7 As shown, the vertical component branch includes a vertical component convolution filter flow network, a third skip path and a third multiplication module. The vertical component sub-band image is input into the vertical component convolution filter flow network and the third skip path respectively. The output end of the vertical component convolution filter flow network and the output end of the third skip path are both connected to the input end of the third multiplication module. The third multiplication module outputs the vertical component sub-band reconstructed image.

[0075] The diagonal component branches, such as Figure 7 As shown, the diagonal component branch includes a diagonal component convolution filter flow network, a fourth skip path and a fourth multiplication module, the diagonal component sub-band image is input into the diagonal component convolution filter flow network and the fourth skip path respectively, the output end of the diagonal component convolution filter flow network and the output end of the fourth skip path are both connected to the input end of the fourth multiplication module, and the fourth multiplication module outputs the diagonal component sub-band reconstructed image.

[0076] The image reconstruction process for the sub-band reconstruction of the three high-frequency components of horizontal component, vertical component, diagonal component and diagonal component includes:

[0077] First, the high-frequency component convolution filter flow network R p (·) Input high-frequency component subband image Processing to obtain high frequency component filter flow chart

[0078] Finally, the high-frequency component subband image is introduced into the corresponding multiplication module and the high-frequency component filter flow graph by using the jump path of the high-frequency component to perform multiplication operations of different dimensions of the broadcast mechanism to obtain the high-frequency component subband reconstructed image.

[0079]

[0080] In this embodiment, preferably, Figure 8As shown, the approximate low-frequency component convolution filter flow network includes a first branch and a second branch connected in parallel, and a series-connected summing unit and a first multi-layer convolution network including multi-layer convolutions, a first convolution neural network is provided on the first branch, a second convolution neural network is provided on the second branch, the depth of the second convolution neural network is less than the depth of the first convolution neural network, the output end of the first convolution neural network and the output end of the second convolution neural network are respectively connected to the input end of the summing unit, and the output end of the summing unit is connected to the input end of the first multi-layer convolution network. Specifically, the depth refers to the number of layers of the convolution neural network.

[0081] The low-frequency part usually contains the characteristics of the signal. A convolutional filter flow network is constructed, and two paths (a first branch and a second branch) are selected to learn pixel-level filters. The first convolutional neural network on the first branch is a deep neural network, preferably but not limited to an 18-layer deep convolutional neural network improved based on Resnet18, wherein the first convolutional layer has a large receptive field of a 7×7 convolution kernel, and can estimate the pixel filter by considering context information.

[0082] The second convolutional neural network on the second branch is shallower than the first convolutional neural network. A shallow convolutional neural network is selected to protect the spatial information of the low-frequency subband from loss by maintaining the spatial resolution of the low-frequency subband image. Finally, the information of the two paths is integrated to estimate the pixel filter. The second convolutional neural network is preferably, but not limited to, a relatively shallow convolutional neural network with full resolution, including multiple cascaded processing networks, the processing network includes a convolutional layer, an activation layer, and a batch normalization layer connected in series in sequence. The second path protects the spatial information of the original image from loss by maintaining the resolution of the original image. Finally, the first image and the second image output by the two paths are merged and spliced ​​by a summing unit. The merged and spliced ​​image is input into the first multi-layer convolutional network. The first multi-layer convolutional network preferably, but not limited to, includes 3 layers of convolution. The first multi-layer convolutional network preferably, but not limited to, includes a second convolutional layer, a second activation layer, a second batch normalization layer, a third convolutional layer, a third activation layer, a third batch normalization layer, and a fourth convolutional layer connected in sequence. Preferably, a mapping layer is also included after the summing unit, and the mapping layer is preferably, but not limited to, a Softmax classifier.

[0083] In this embodiment, preferably, the horizontal component convolution filter flow network includes a third convolutional neural network and a second multi-layer convolutional network including multi-layer convolutions connected in sequence, and the depth of the third convolutional neural network is greater than the depth of the second convolutional neural network.

[0084] In this embodiment, preferably, the vertical component convolution filter flow network includes a fourth convolutional neural network and a third multi-layer convolutional network including multi-layer convolutions connected in sequence, and the depth of the fourth convolutional neural network is greater than the depth of the second convolutional neural network.

[0085] In this embodiment, preferably, the diagonal component convolution filter flow network includes a fifth convolutional neural network and a fourth multi-layer convolutional network including multi-layer convolutions connected in sequence, and the depth of the fifth convolutional neural network is greater than the depth of the second convolutional neural network.

[0086] In this embodiment, the high-frequency part of the image contains not only noise, but also usually contains signal details or difference information. In order to make the convolution filter flow select this information specifically, the BN layer can promote the model to use the input high-frequency component information as efficiently as possible, thereby improving the model performance. The third convolutional neural network, the fourth convolutional neural network and the fifth convolutional neural network use the same network structure as the first convolutional neural network, that is, an 18-layer deep convolutional neural network improved based on Resnet18, which will not be repeated here. ReLU activation layers and BN layers are added after multiple convolutional layers, and finally the estimated pixel-level filter is obtained through softmax mapping.

[0087] In this embodiment, the image compression sensing system provided in this embodiment is an end-to-end self-supervisory framework. The input and labels required for training are obtained by inputting the original image I 2 Obtained, the dataset training pair is represented as i=1,...,N', N' represents the number of training pairs in the training data set, represents the i-th original image, represents the final reconstructed image corresponding to the i-th original image, using the mean absolute error (MAE) as the cost function, striving to obtain the initial reconstructed image And reconstruct the image As close as possible to the original image I 2 , so constraints are imposed on both the initial reconstructed image and the reconstructed image during network training. Therefore, the loss function in the training of the image compression sensing system is:

[0088] in, Represents the initial reconstructed image corresponding to the i-th original image.

[0089] In one application scenario of the present invention, a training set of a sampling module to a deep reconstruction network is constructed. The training set is preferably but not limited to DIV2K_train_HR, which is a commonly used data set in the fields of image super-resolution, image denoising, and compressed sensing reconstruction. It has 800 images with a higher resolution. Each training image is cropped and selected with a step size of 128 pixels to select 61,680 128×128 image blocks as a training set. The test set includes commonly used standard data sets such as Set5, Set11, and BSD68. The network based on discrete wavelet transform and convolutional filter flow model (DWT-CFFNet) proposed in the present invention is different from other image-to-image regression convolutional neural networks. The model uses the Adam optimization algorithm during the training process, with 200 epochs, a batch size of 32, and a learning rate of 10. -4 , the filter size is set to 17, then DWT-CFFNet outputs 17×17=289 weights at each pixel position of the wavelet component image. All experiments were completed on an Intel Corei3-10100CPU, 3.6GHz CPU processor, NVIDIAGeForce GTX 1080 Ti graphics card and Pycharm on the Ubuntu18.04.6 LTS operating system. The evaluation criteria selected subjective visual effect display and objective image quality evaluation indicators Peak Signal to Noise Ratio (PSNR) and Structure Similarity (SSIM).

[0090] In this application scenario, the compressed sensing reconstruction method provided by the present invention, the existing MH algorithm, ReconNet, CSNet, and ISTANet+ four compressed sensing reconstruction algorithms are compared, and it is found that when the test image is at three sampling rates of 10%, 20%, and 30%, the reconstruction quality of the DWT-CFFNet proposed by the present invention is the best, especially at the low sampling rate (10% sampling rate), while ReconNet has the worst reconstruction quality among several algorithms, ISTA-Net+ has better reconstruction quality when the sampling rate is higher (20% sampling rate, 30% sampling rate), and CSNet has better reconstruction quality when the sampling rate is lower (10% sampling rate). Compared with the MH algorithm, ReconNet, CSNet, and ISTANet+, the average PSNR of the DWT-CFFNet proposed in the present invention is improved by 3.05dB, 3.40dB, 0.90dB, and 1.72dB at a sampling rate of 10%, 3.08dB, 4.94dB, 0.55dB, and 1.03dB at a sampling rate of 20%, and 2.30dB, 4.82dB, 0.08dB, and 0.74dB at a sampling rate of 30%. The DWT-CFFNet proposed in the present invention can restore more texture details and the image quality is clearer, which is based on the fact that it takes into account both the global context and the local neighborhood information of the pixel.

[0091] In this application scenario, Fig.10 The reconstructed images of the four wavelet components in the deep reconstruction network of the DWT-CFFNet proposed in the present invention and their corresponding learned pixel-level filter flows are shown at a sampling rate of 20%. In the experiment, in order to show the spatial variation structure of the filter, 2Dt-SNE is used to assign a color to each centroid, and the nearest centroid is visualized at the position of each filter. The distribution of the filter flow vector is visualized by calculating the expected filter flow vector, showing that the different colors in the filter flow visualization diagram are the changes in the corresponding pixels at the corresponding spatial positions, and the depth of the color indicates the intensity of the change.

[0092] The present invention proposes an image compressed sensing reconstruction network model based on discrete wavelet transform filter flow. The model uses convolutional layers to perform compressed sampling process and initial reconstruction process, and jointly optimizes the measurement matrix and convolution filter flow. It not only uses discrete wavelet transform to take into account the high and low frequency component information of the image, but also combines the wavelet domain with the pixel domain, and considers the local neighborhood information of the image pixels, which effectively improves the effect of compressed sensing image reconstruction and has good interpretability.

[0093] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0094] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. Image compression sensing method based on wavelet transform and convolution filter flow, characterized in that: include: Obtaining multiple image patch measurements; Performing initial reconstruction on the measurement values ​​of the multiple image blocks to obtain an initial reconstructed image; Inputting the initial reconstructed image into a deep reconstruction network for processing to obtain a final reconstructed image, wherein the deep reconstruction network includes a discrete wavelet transform network, a pixel-level filter network based on a convolution filter flow, and an inverse discrete wavelet transform fusion network connected in sequence; Wherein, the deep reconstruction network performs: Performing discrete wavelet transform on the initial reconstructed image to obtain an approximate low-frequency component sub-band image, a horizontal component sub-band image, a vertical component sub-band image and a diagonal component sub-band image; The approximate low-frequency component sub-band image, the horizontal component sub-band image, the vertical component sub-band image and the diagonal component sub-band image are reconstructed into an approximate low-frequency component sub-band reconstructed image, a horizontal component sub-band reconstructed image, a vertical component sub-band reconstructed image and a diagonal component sub-band reconstructed image respectively using corresponding pixel-level filters based on convolution filter flows; The approximate low-frequency component sub-band reconstructed image, the horizontal component sub-band reconstructed image, the vertical component sub-band reconstructed image and the diagonal component sub-band reconstructed image are respectively subjected to inverse discrete wavelet transform processing, and the images after inverse discrete wavelet transform processing are fused to obtain the final reconstructed image; The final reconstructed image is obtained by fusing the image processed by inverse discrete wavelet transform. for: Where (u, v) represents the pixel position; the size of the reconstructed image is the same as the original image, both of which are M × N; represents the approximate low-frequency component subband reconstructed image, Represents the wavelet function of the approximate low-frequency component; component index p = {H, V, D}, when index p = H, represents the horizontal component subband reconstructed image, ψ H (u, v) represents the wavelet function of the horizontal component; when index p = V, represents the vertical component subband reconstructed image, ψ V (u, v) represents the wavelet function of the vertical component; when index p = D, represents the diagonal component subband reconstructed image, ψ D (u, v) represents the wavelet function of the diagonal component.

2. The image compression sensing method based on wavelet transform and convolution filter flow according to claim 1, characterized in that: The step of obtaining the measurement values ​​of the multiple image blocks is specifically as follows: The original image is input into a sampling network to obtain a plurality of image block measurement values, wherein the sampling network includes a sampling convolutional layer.

3. The image compression sensing method based on wavelet transform and convolution filter flow as claimed in claim 1, characterized in that: The step of initially reconstructing the measurement values ​​of the multiple image blocks to obtain the initial reconstructed image specifically includes: Input multiple image block measurement values ​​into the initial reconstruction convolution layer to obtain a vector representation of the image block; The vector representations of multiple image blocks are reshaped into an initial reconstructed image with the same size as the original image through a pixel shuffling operation.

4. Image compression sensing system based on wavelet transform and convolution filter flow, characterized in that: include: A sampling module, used for compressing and sampling the original image to obtain a plurality of image block measurement values; An initial reconstruction module, used for performing initial reconstruction on a plurality of image block measurement values ​​to obtain an initial reconstructed image; A deep reconstruction network, used to further reconstruct the initial reconstructed image to obtain a final reconstructed image, wherein the deep reconstruction network includes a discrete wavelet transform network, a pixel-level filter network based on a convolution filter flow, and an inverse discrete wavelet transform fusion network connected in sequence; Wherein, the deep reconstruction network performs: Performing discrete wavelet transform on the initial reconstructed image to obtain an approximate low-frequency component sub-band image, a horizontal component sub-band image, a vertical component sub-band image and a diagonal component sub-band image; The approximate low-frequency component sub-band image, the horizontal component sub-band image, the vertical component sub-band image and the diagonal component sub-band image are reconstructed into an approximate low-frequency component sub-band reconstructed image, a horizontal component sub-band reconstructed image, a vertical component sub-band reconstructed image and a diagonal component sub-band reconstructed image respectively using corresponding pixel-level filters based on convolution filter flows; The approximate low-frequency component sub-band reconstructed image, the horizontal component sub-band reconstructed image, the vertical component sub-band reconstructed image and the diagonal component sub-band reconstructed image are respectively subjected to inverse discrete wavelet transform processing, and the images after inverse discrete wavelet transform processing are fused to obtain the final reconstructed image; The final reconstructed image is obtained by fusing the image processed by inverse discrete wavelet transform. for: Where (u, v) represents the pixel position; the size of the reconstructed image is the same as the original image, both of which are M × N; represents the approximate low-frequency component subband reconstructed image, Represents the wavelet function of the approximate low-frequency component; component index p = {H, V, D}, when index p = H, represents the horizontal component subband reconstructed image, ψ H (u, v) represents the wavelet function of the horizontal component; when index p = V, represents the vertical component subband reconstructed image, ψ V (u, v) represents the wavelet function of the vertical component; when index p = D, represents the diagonal component subband reconstructed image, ψ D (u, v) represents the wavelet function of the diagonal component.

5. The image compression sensing system based on wavelet transform and convolution filter flow as claimed in claim 4, characterized in that: The discrete wavelet transform network performs discrete wavelet transform on the initial reconstructed image to obtain an approximate low-frequency component sub-band image, a horizontal component sub-band image, a vertical component sub-band image and a diagonal component sub-band image.

6. The image compression sensing system based on wavelet transform and convolution filter flow as claimed in claim 5, characterized in that: The pixel-level filter network comprises: The approximate low-frequency component branch includes an approximate low-frequency component convolution filter flow network, a first skipping path and a first multiplication module, wherein the approximate low-frequency component subband image is input into the approximate low-frequency component convolution filter flow network and the first skipping path respectively, the output end of the approximate low-frequency component convolution filter flow network and the output end of the first skipping path are both connected to the input end of the first multiplication module, and the first multiplication module outputs the approximate low-frequency component subband reconstructed image; The horizontal component branch includes a horizontal component convolution filter flow network, a second skipping path, and a second multiplication module. The horizontal component sub-band image is respectively input into the horizontal component convolution filter flow network and the second skipping path. The output end of the horizontal component convolution filter flow network and the output end of the second skipping path are both connected to the input end of the second multiplication module. The second multiplication module outputs a horizontal component sub-band reconstructed image. A vertical component branch includes a vertical component convolution filter flow network, a third skipping path and a third multiplication module, wherein the vertical component sub-band image is input into the vertical component convolution filter flow network and the third skipping path respectively, the output end of the vertical component convolution filter flow network and the output end of the third skipping path are both connected to the input end of the third multiplication module, and the third multiplication module outputs a vertical component sub-band reconstructed image; The diagonal component branch includes a diagonal component convolution filter flow network, a fourth skip path and a fourth multiplication module. The diagonal component sub-band images are respectively input into the diagonal component convolution filter flow network and the fourth skip path. The output end of the diagonal component convolution filter flow network and the output end of the fourth skip path are both connected to the input end of the fourth multiplication module. The fourth multiplication module outputs the diagonal component sub-band reconstructed image.

7. The image compression sensing system based on wavelet transform and convolution filter flow according to claim 6, characterized in that: The approximate low-frequency component convolution filter flow network includes a first branch and a second branch connected in parallel, and a series-connected summing unit and a first multi-layer convolution network including multi-layer convolutions, wherein the first branch is provided with a first convolutional neural network, the second branch is provided with a second convolutional neural network, the depth of the second convolutional neural network is less than the depth of the first convolutional neural network, the output end of the first convolutional neural network and the output end of the second convolutional neural network are respectively connected to the input end of the summing unit, and the output end of the summing unit is connected to the input end of the first multi-layer convolutional network; And / or, the horizontal component convolution filter flow network includes a third convolutional neural network and a second multi-layer convolutional network including multi-layer convolutions connected in sequence, and the depth of the third convolutional neural network is greater than the depth of the second convolutional neural network; And / or, the vertical component convolution filter flow network includes a fourth convolutional neural network and a third multi-layer convolutional network including multi-layer convolutions connected in sequence, and the depth of the fourth convolutional neural network is greater than the depth of the second convolutional neural network; And / or, the diagonal component convolution filter flow network includes a fifth convolutional neural network and a fourth multi-layer convolutional network including multi-layer convolutions connected in sequence, and the depth of the fifth convolutional neural network is greater than the depth of the second convolutional neural network.

8. The image compression sensing system based on wavelet transform and convolution filter flow according to any one of claims 4 to 7, characterized in that: The training pair for training the image compression sensing system is represented as i = 1, ..., N', N' represents the number of training pairs in the training data set, represents the i-th original image, represents the final reconstructed image corresponding to the i-th original image; the loss function in the training of the image compression sensing system is: in, Represents the initial reconstructed image corresponding to the i-th original image.

Images