Image compression sensing method and system based on fast Fourier convolution and convolution filter flow

Abstract

The present invention provides an image compression sensing method and system based on fast Fourier convolution and convolution filter flow. The image compression sensing method includes: obtaining multiple image block measurement values; performing initial reconstruction on the multiple image block measurement values ​​to obtain an initial reconstructed image; further reconstructing the initial reconstructed image based on fast Fourier convolution to obtain a Fourier convolution reconstructed image; inputting the Fourier convolution reconstructed image into a pixel-level filter based on a convolution filter flow network to obtain a filtering result, and convolving the filtering result with the Fourier convolution reconstructed image to obtain a final reconstructed image. Fast Fourier convolution further reconstructs the initial reconstructed image, and the fast Fourier convolution receptive field can cover the entire image, effectively utilizing global context information, and then obtaining local information of the local neighborhood of pixels through a pixel-level filter. The combination of the two effectively improves the compressed sensing image reconstruction effect and visual effect.

Description

Technical Field

[0001] The present invention belongs to the technical field of image compression sensing, and in particular relates to an image compression sensing method and system based on fast Fourier convolution and convolution filter flow. Background Art

[0002] In the study of image compressed sensing reconstruction algorithms, some traditional algorithms such as MH (such as the MH algorithm published in the paper "Image compressed sensing using convolutional neural network[J]" published by W. Shi, F. Jiang, S. Liu, etc. in IEEE Transactions on Image Processing in 2020) mostly start from image priors and add additional optimization methods or combine iterative threshold algorithms. Most of them have high computational complexity and poor image reconstruction quality. Deep learning-based algorithms such as ReconNet (such as the paper "ReconNet: Non-iterative reconstruction of images from compressively sensed measurements[C]" published by K. Kulkarni, S. Lohit, P. Turaga, etc. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) in 2016, which has a public ReconNet algorithm) have low computational costs, but most of them use small receptive fields, resulting in the loss of non-local information of the image itself, low reconstruction quality, and the problem of poor interpretability of deep learning methods has not been solved. Summary of the invention

[0003] The present invention aims to at least solve the technical problems existing in the prior art and provide an image compression sensing method and system based on fast Fourier convolution and convolution filter flow.

[0004] In order to achieve the above-mentioned purpose of the present invention, according to the first aspect of the present invention, the present invention provides an image compression sensing method based on fast Fourier convolution and convolution filter flow, including: obtaining multiple image block measurement values; performing initial reconstruction on the multiple image block measurement values ​​to obtain an initial reconstructed image; further reconstructing the initial reconstructed image based on fast Fourier convolution to obtain a Fourier convolution reconstructed image; inputting the Fourier convolution reconstructed image into a pixel-level filter based on a convolution filter flow network to obtain a filtering processing result, and convolving the filtering processing result with the Fourier convolution reconstructed image to obtain a final reconstructed image.

[0005] In order to achieve the above-mentioned purpose of the present invention, according to the second aspect of the present invention, the present invention provides an image compression sensing system based on fast Fourier convolution and convolution filter flow, comprising: a sampling module, used for compressing and sampling the original image to obtain multiple image block measurement values; an initial reconstruction module, used for initially reconstructing the multiple image block measurement values ​​to obtain an initial reconstructed image; a Fourier convolution reconstruction module, further reconstructing the initial reconstructed image based on fast Fourier convolution to obtain a Fourier convolution reconstructed image; a pixel-level filter, filtering the Fourier convolution reconstructed image, wherein the pixel-level filter adopts a convolution filter flow network architecture;

[0006] The fusion module convolves the filtering processing result of the pixel-level filter with the Fourier convolution reconstructed image to obtain the final reconstructed image.

[0007] The above-mentioned technical solution: After the initial reconstruction of the image, the present invention further reconstructs the initial reconstructed image based on fast Fourier convolution. The fast Fourier convolution has a non-local receptive field, and the receptive field can cover the entire image, effectively utilizing the global context information, which helps to improve the quality of the reconstructed image. The local information of the local neighborhood of the pixel is then obtained through a pixel-level filter, which has good interpretability. The combination of fast Fourier convolution and convolution filter flow fully utilizes the sparsity of the signal and the structural information of the signal itself, effectively improving the compressed sensing image reconstruction effect and visual effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a flowchart of an image compression sensing method in a preferred embodiment of the present invention;

[0009] Figure 2 It is a detailed flow chart of an image compression sensing method in a preferred embodiment of the present invention;

[0010] Figure 3 is a structural block diagram of an image compression sensing system in a preferred embodiment of the present invention;

[0011] Figure 4 It is a schematic diagram of the structure of a sampling module in a preferred embodiment of the present invention;

[0012] Figure 5 is a schematic diagram of a pixel shuffling process in a preferred embodiment of the present invention;

[0013] Figure 6 is a schematic diagram of the structure of an initial reconstruction module in a preferred embodiment of the present invention;

[0014] Figure 7 It is a schematic diagram of the structure of a fast Fourier convolution network in a preferred embodiment of the present invention;

[0015] Figure 8 It is a structural schematic diagram of a Fourier convolution reconstruction module in a preferred embodiment of the present invention;

[0016] Fig. 9 It is a final reconstructed image and filter flow graph of an image compression sensing system at a 30% sampling rate in a preferred implementation manner of the present invention. Description of the drawings:

[0018] C convolution layer; B batch normalization layer (BN); R activation function layer; F real part fast Fourier transform layer; IF real part inverse fast Fourier transform layer; U upsampling layer; M maximum pooling layer. DETAILED DESCRIPTION

[0019] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0020] In the description of the present invention, it is necessary to understand that the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0021] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0022] As the core of deep learning, convolutional neural networks are the main driving force for the great progress in some research fields in recent years. The core of convolutional neural networks is the receptive field, which is the area of ​​the input image of the convolutional neural network feature map. Most modern convolutional neural networks use convolutional layers with small receptive fields. This local operation cannot achieve long-distance dependence of feature extraction when the number of layers is small. The receptive field can only be increased by stacking convolutional layers to linearly or exponentially increase the size of the receptive field, or by using dilated convolutions. However, for the image reconstruction problem in compressed sensing, it is more sensitive to contextual information, and a small receptive field is relatively unfavorable for image reconstruction, because a larger receptive field plays an important role in understanding the global structure of the image and solving compressed sensing image reconstruction.

[0023] Fast Fourier Convolution (FFC) is a new type of convolution module proposed recently. For example, the paper "Image compressed sensing using convolutional neural network[J]" published by W. Shi, F. Jiang, S. Liu, etc. was published in the 2020 journal IEEE Transactions on Image Processing, 2020, 29: 375-388. The paper proposed fast Fourier convolution, which has the advantages of non-local receptive field and cross-scale feature fusion within the convolution unit. According to the frequency domain convolution theorem in Fourier theory, only one point in the frequency domain needs to be changed to affect the global features in the spatial domain. Fast Fourier convolution allows the receptive field within the image range and can cover the entire image. The inventors of the present application found that it can even be used in some early convolution layers in the image reconstruction stage.

[0024] This application discloses a fast Fourier convolutional network structure, such as Figure 7 As shown in Figure 1. The fast Fourier convolution network consists of two parallel paths, one is a local path that performs ordinary convolution on a part of the input feature channel, and the other is a global path that operates in the spectral domain. Each path can capture information with different receptive fields and complement each other, and the information exchange between paths is performed internally. The input image X is split into X = {X l , X g}, X l For local path learning, X g Used to capture global context information in the global path.

[0025] In the fast Fourier convolutional network structure, Figure 7As shown, the local path includes two parallel branches, each of which has a convolution layer. The processed images output by the two branches are element-wise summed and then passed through a batch normalization layer and an activation function layer to obtain the local tensor Y l The global path also includes two parallel branches, one of which is provided with a convolution layer, and the other branch is provided with a first global convolution layer, a first global batch normalization layer, a first global activation layer, a global residual network, and a second global convolution layer in sequence. The global residual network includes a real fast Fourier transform layer, a second global convolution layer, a second global batch normalization layer, a second global activation layer, a real inverse fast Fourier transform layer, a global element summation unit, and a jumper connected in sequence. The jumper introduces the output image of the first global activation layer into the global element summation unit to perform element summation with the output image of the real inverse fast Fourier transform layer, and then inputs the summation result into a batch normalization layer and an activation layer in sequence to obtain the global tensor Y g , fused local tensor Y l and the global tensor Y g Get the Fast Fourier Convolutional Network output image.

[0026] Suppose the input feature map of a fast Fourier convolutional network is Among them, H×W is the spatial resolution, C is the number of feature channels, first split along the dimension of the number of feature channels into X={X l , X g}, local part is used for local path learning, is used to capture the global context, α in ∈[0, 1], α in Indicates the percentage of channels that the input is assigned to the global path. Output tensor The corresponding Y={Y l , Y g There are local segmentation output and global segmentation output in it, and the output tensor dimension is also determined by a hyperparameter α out ∈[0, 1], α out Represents the percentage of channels that the output is assigned to the global path. The process of fast Fourier convolution update can be expressed as:

[0027] Y l =Y l→l +Y g→l =f l (X l )+f g→l (X g );

[0028] Y g =Y g→g +Y l→g =f g (Xg )+f l→g (X l );

[0029] For Y, which aims to capture small-scale information l→l , component Y converted through paths g→l and Y l→g Both use ordinary convolution, Y g→g The purpose of the spectral converter is to convert the original spatial features into the spectral domain, a discrete Fourier transform using an accelerated version of the Cooley-Tukey algorithm, effectively extending the receptive field of the convolution to the full resolution of the input feature map. Figure 7 As shown, a real fast Fourier transform (real FFT) is used in the global path to interpret the global context, which only uses half of the spectrum, and an inverse transform is applied to restore the spatial structure to ensure that the output is real-valued. Finally, the global path and the local path are fused and output.

[0030] In the theory of image compression sensing, most modern techniques for studying image inverse problems focus on regularization techniques for computational optimization, effectively utilizing various image priors to obtain better image reconstruction effects, but this results in expensive computational costs. At the same time, the compressed sensing image algorithm based on deep learning effectively utilizes data-driven methods to make image reconstruction faster, but it lacks the guarantee of interpretability and robustness. Inspired by the idea of ​​convolutional filter flow in the prior art, such as the filter flow idea disclosed in the paper "Filter flow" published by SM Seitz, S. Baker et al. at the 2009 IEEE 12th International Conference on Computer Vision. Kyoto in 2009, this application proposes a new image compression sensing convolutional filter flow framework. The convolutional filter flow (CFF) not only retains good interpretability and control over the reconstruction results, but also ensures fast computational reasoning. It provides a new solution for such low-level image processing tasks in compressed sensing.

[0031] In the convolution filter flow, the pixels in the local neighborhood of the input image are linearly combined to reconstruct the pixels at the corresponding position in the output image. The difference from ordinary convolution is that the weight of the filter varies with the spatial position. This application estimates the weight of the regression filter for a given input image. In short, the convolution filter flow does not focus on the flow between the original image and the undersampled image, but focuses on the entire convolution filter flow and the transformation of the image. Although the convolution filter flow is based on a convolutional neural network, it is different from other compressed sensing reconstruction algorithms based on deep learning, because the convolution filter flow can show an explicit representation from the input image to the output reconstructed image, and how it is transformed is intuitive, transparent and explainable.

[0032] Specifically, according to the filtering operation I′(u, v) represents an image, u = (u, v) is the position of the pixel, and on each pixel of the image, a filter can be defined Acts on the image region centered at pixel u:

[0033]

[0034] Among them, i, j control the size of the neighborhood. The convolution filter flow models the image transformation I′1→I′2 as a linear mapping. As described above, each output pixel depends only on the local neighborhood of the corresponding input pixel. Such a flow can be solved by the convolution filter flow framework as a constrained linear system through learning methods.

[0035] I′2=TI′1,T∈τ (2)

[0036] Where T is a vector matrix, each row of which defines a linear combination of pixels in I′1, acting on the vector matrix of the input image, generating pixels at appropriate positions in I′2. T∈Γ is a placeholder for the entire set of additional constraints. For example, T corresponding to ordinary convolution is usually a circulant matrix, whose rows are circulant permutations of a single set of filter weights, which are usually constrained to have compact local non-zeros. Equation (2) seems simplified, but it is not considered to solve it directly in theory. For example, when the image size is different, the filter needs to be changed in space, which requires a learnable method.

[0037] In order to better estimate the transformation between the input image and the output image, the convolution filter stream trains a learnable function f by learning. w (·), w is the hyperparameter of the convolutional filter flow network. The convolutional filter flow model can be expressed as:

[0038]

[0039] Need to give an image pair i = 1, ..., N'. In the theory of compressed sensing, the degraded image is taken as I'1 and the original image is taken as I'2 to form an image training pair to find the optimal parameter w. In the design of the loss function, we strive to minimize the reconstructed image The difference between the original image I′2 and the original image I′2, the loss function is set as:

[0040]

[0041] By design w The structure of (·) is used as a hard constraint to select an appropriate regularization term. As a soft constraint. Each pixel in I′1 depends only on the local neighborhood centered on the corresponding pixel, which is equivalent to introducing a structural constraint that each pixel of the output depends only on the local neighborhood of the corresponding input pixel. The size of this neighborhood is also a hyperparameter.

[0042] The structural constraint of the local neighborhood of the input pixel is implemented through the "im2col" operation, which actually optimizes the convolution operation by converting the convolution operation between matrices into the multiplication operation between matrices. This is based on the similarity between the convolution multiplication and addition operation and the positive operation within the vector, which reduces the time and space costs and solves the problem of discontinuous memory reading caused by following the mathematical steps of the convolution operation.

[0043] Convolutional filter flow is a completely self-supervised learning framework. As an "ideal state" of deep learning, the convolutional filter flow model does not require manually labeled data and can be learned directly from unlabeled data. Given an original image block, it is compressed and sampled, and initially reconstructed to obtain an initial reconstructed image block, which forms a training pair with the original image block, and the model parameters are trained in a self-supervised manner.

[0044] The present invention also discloses an image compression sensing method based on fast Fourier convolution and convolution filter flow, such as Figure 1 As shown, including:

[0045] Step S1, obtaining a plurality of image block measurement values. The plurality of image block measurement values ​​are a plurality of image block measurement values ​​obtained after compression sampling of the image to be compressed sensing.

[0046] Step S2: performing initial reconstruction on the measurement values ​​of the multiple image blocks to obtain an initial reconstructed image.

[0047] Step S3, further reconstructing the initial reconstructed image based on fast Fourier convolution to obtain a Fourier convolution reconstructed image.

[0048] Step S4, inputting the Fourier convolution reconstructed image into a pixel-level filter based on a convolution filter flow network to obtain a filtering result, and convolving the filtering result with the Fourier convolution reconstructed image to obtain a final reconstructed image.

[0049] In this embodiment, in order to reduce the computational complexity of the whole network based on fast Fourier convolution and convolution filter flow, the block-based compressed sensing (BCS) algorithm is used to perform block operations on the original image. In order to improve the sampling adaptability, Figure 4 As shown, step S1 specifically includes: inputting the original image into the sampling network to obtain multiple image block measurement values, the sampling network includes a sampling convolution layer, and the sampling convolution layer includes n s The size of the sampling convolution kernel is S×S×k. Block compressed sensing uses a size of n s × 2 The measurement matrix is ​​denoted as A s , for image block I i For example, the compressed sampling process can be expressed as i =A s I i The specific sampling process of this application is to transform the measurement matrix A s Each row in is regarded as a convolution kernel, and an unbiased convolution layer is used to simulate the block compression sampling process. Then, when the sampling rate is M / N, each convolution kernel of size S×S×k is used, because the image block I i The size is also S×S×k, so the measurement matrix A s There are n s Row, where n s =MkS 2 / N, that is, n can be measured from an image block s The compressed sampling process can be expressed as:

[0050] y=S(I)=W s *I, where S(·) represents the compressed sampling operation, y represents the compressed sensing measurement value, and W s is the convolution kernel, whose size is S×S×k and has a total of n s , * represents the convolution operation, I is the input image to be measured, and the sampling network can be expressed as Figure 4 Training the measurement matrix through learning can more effectively utilize the characteristics of the image itself, such as structural information, and lay a good foundation for the subsequent reconstruction process.

[0051] In this embodiment, in order to convert the image block observation value into an image with the same size as the original image for subsequent further reconstruction, preferably, as Figure 6As shown, step S2 specifically includes:

[0052] Input multiple image block measurement values ​​into the initial reconstruction convolution layer to obtain the vector representation of the image block; Figure 5 As shown, the vector representations of multiple image blocks are reshaped into an initial reconstructed image with the same size as the original image through a pixel shuffling operation.

[0053] In this embodiment, in the initial reconstruction network of the present application, the input is the measured value y after the compressed sampling of the output of the sampling network. Similar to the sampling network, a convolution layer is first used to extract the information of the measured value and transform it into image information. The specific method is as follows: First, Set the convolution kernel of a convolution layer to a size of 1×1×n s , and there are kS 2 filters, because the measurement value y is also 1×1×n s , each image block contains kS 2 elements, the output is 1×1×kS 2 The output is a tensor of size , which is the vector representation of each initial reconstructed image block. Then, as Figure 5 As shown, the pixel shuffle operation converts 1×1×kS 2 The tensor output is reshaped into S×S×k, with a total of n s image blocks, ensuring that the initial reconstructed image is consistent with the original image size, and can achieve tensor splicing and shaping.

[0054] In this implementation, in order to effectively extend the receptive field of the convolution to the full resolution of the input feature map, Figure 8 As shown, step S3 specifically includes:

[0055] The initial reconstructed image is sequentially input into the fast Fourier convolution network, multiple fast Fourier convolution residual networks and the first convolution layer to obtain the global feature image; the initial reconstructed image and the global feature image are element-wise summed to obtain the Fourier convolution reconstructed image. In the early stage, the global context information is extracted through the fast Fourier convolution network, and the parameter is set to α in =0,α out = 0.5, and the convolution kernel size is 3 × 3 × 64. Multiple fast Fourier convolution residual networks will effectively expand the receptive field of the convolution to the full resolution of the input feature map, preferably but not limited to 5 fast Fourier convolution residual networks, and the last layer uses the first convolution layer of 1 × 1 standard convolution to splice the feature map and the initial reconstructed image into image information to obtain the Fourier convolution reconstructed image.

[0056] In this embodiment, through the processing of step S3, the Fourier convolution reconstructed image outputted is a relatively good degraded image, and has the conditions for learning the pixel-level filter in the subsequent convolution filter flow.

[0057] In this embodiment, in order to make the pixel-level filter focus not only on local information but also on global context information, so as to improve the reconstruction quality. Figure 3 As shown, the pixel-level filter includes a first convolutional neural network, a second convolutional neural network, a third convolutional neural network and a mapping layer; the Fourier convolution reconstructed image is input into the first convolutional neural network and the second convolutional neural network to obtain the first image and the second image respectively, the Fourier convolution reconstructed image is input into the first convolutional neural network to obtain the first image, and the Fourier convolution reconstructed image is input into the second convolutional neural network to obtain the second image; the first image and the second image are element-summed and the summation results are sequentially input into the third convolutional neural network and the mapping layer for processing to obtain the filtering processing result; the depth of the first convolutional neural network is greater than the depth of the second convolutional neural network, and specifically, the depth refers to the number of layers of the convolutional neural network. The third convolutional neural network includes multiple layers of convolution, preferably but not limited to 3 layers of convolution.

[0058] In this embodiment, if Figure 3 As shown, the pixel-level filter is divided into two parallel paths, wherein the first convolutional neural network is the first path and the second convolutional neural network is the second path. The first path is preferably, but not limited to, an 18-layer deep convolutional neural network improved based on Resnet18, which has a large receptive field and can estimate the pixel filter by considering contextual information. The second path is preferably, but not limited to, a relatively shallow convolutional neural network with full resolution, including multiple cascaded processing networks, the processing network including convolutional layers, activation layers and batch normalization layers connected in series in sequence, and the second path protects the spatial information of the original image from loss by maintaining the resolution of the original image. Finally, the first image and the second image output by the two paths are merged and spliced. The merged and spliced ​​image is input into the third convolutional neural network, and the third convolutional neural network preferably, but not limited to, includes a second convolutional layer, a second activation layer, a second batch normalization layer, a third convolutional layer, a third activation layer, a third batch normalization layer, and a fourth convolutional layer connected in sequence. The mapping layer is preferably, but not limited to, a Softmax classifier. The process of the pixel-level filter can be described as:

[0059]

[0060] in, represents the final reconstructed image, Reconstruct the image for Fourier convolution, f w (·) represents the filter function learned by the pixel-level filter, R c(·) represents a convolutional filter flow network.

[0061] The present invention also discloses an image compression sensing system of fast Fourier convolution and convolution filtering flow, such as Figure 2 As shown, in a preferred embodiment, the system includes: a sampling module, which is used to compress and sample the original image to obtain multiple image block measurement values; an initial reconstruction module, which is used to perform initial reconstruction on the multiple image block measurement values ​​to obtain an initial reconstructed image; a Fourier convolution reconstruction module, which further reconstructs the initial reconstructed image based on fast Fourier convolution to obtain a Fourier convolution reconstructed image; a pixel-level filter, which performs filtering processing on the Fourier convolution reconstructed image, and the pixel-level filter adopts a convolution filter flow network architecture; a fusion module, which convolves the filtering processing result of the pixel-level filter with the Fourier convolution reconstructed image to obtain a final reconstructed image.

[0062] In this embodiment, preferably, the sampling module includes a sampling convolution layer, such as Figure 4 shown.

[0063] In this embodiment, preferably, the initial reconstruction module includes an initial reconstruction convolution layer and a pixel shuffling unit connected in sequence, such as Figure 6 As shown, the initial reconstruction process can be expressed as:

[0064]

[0065] in, is the initial reconstructed image, R0(·) represents the initial reconstruction operation, PixelShuffle(·) represents the pixel shuffle operation, and W initial kS 2 1×1×n s The size of the convolution kernel.

[0066] In this embodiment, preferably, the Fourier convolution reconstruction module includes a first summing unit, and a fast Fourier convolution network, a plurality of fast Fourier convolution residual networks and a first convolution layer connected in sequence; the input end of the fast Fourier convolution network is connected to the output end of the initial reconstruction module, the output end of the first convolution layer is connected to the first input end of the first summing unit, the output end of the initial reconstruction module is also connected to the second input end of the first summing unit, and the first summing unit outputs the Fourier convolution reconstructed image. The structure of the fast Fourier convolution network FFC is as follows: Figure 7 As shown in Figure 2, the network structure of the Fourier convolution reconstruction module is as follows: Figure 8 shown.

[0067] In this implementation, the Fourier convolution reconstruction module can be expressed as:

[0068]

[0069] in, is the reconstructed image of the fast Fourier convolution residual network, and is also the degraded image of the overall fast Fourier convolution filter flow and convolution filter flow network; R F (·) represents the fast Fourier convolution network, multiple fast Fourier convolution residual networks and the first convolution layer connected in sequence; For the initial reconstructed image, it acts as a residual connection to prevent gradient vanishing during network training, allowing the overall network framework to learn "identical mapping + perturbation" instead of the optimal mapping to improve the reconstruction quality.

[0070] In this embodiment, preferably, Figure 2 As shown, the pixel-level filter includes a first convolutional neural network, a second convolutional neural network, a third convolutional neural network, a second summation unit and a mapping layer, the depth of the first convolutional neural network is greater than the depth of the second convolutional neural network, and the third convolutional neural network includes multiple layers of convolution; the output end of the Fourier convolution reconstruction module is respectively connected to the input end of the first convolutional neural network and the input end of the second convolutional neural network, the output end of the first convolutional neural network and the output end of the second convolutional neural network are both connected to the input end of the second summation unit, the output end of the second summation unit is connected to the input end of the third convolutional neural network, and the output end of the third convolutional neural network is connected to the input end of the mapping layer.

[0071] In this embodiment, a convolution filter flow-based image compression sensing framework is constructed from the sampling module to the fusion module. Given an input image I2, a brief process based on the fast Fourier convolution and convolution filter flow model network is: first, the sampling network S is used to obtain the compressed sensing measurement value y, and then the initial reconstruction network R0 is used to learn the initial reconstructed image from the measurement value. Then the fast Fourier convolution network R in the deep reconstruction network F Get further restored image Finally, from the convolutional filter flow network R c Learn pixel-level filters Then accurately restore the original image The network based on fast Fourier convolution and convolutional filter flow is an end-to-end self-supervised network. The input and labels required for training are obtained through the input image I2. The training pair of the data set can be expressed as i=1,...,N。 Based on the fast Fourier convolution and convolution filter flow network, the mean absolute error (MAE) is used as the cost function to strive for the initial reconstruction image And reconstruct the image Both are as close to the original image I2 as possible, so constraints are imposed on the initial reconstructed image and the reconstructed image during network training, and the loss function in the training process is constructed as follows:

[0072]

[0073] Wherein, N represents the total number of image blocks in the training data set; i represents the image block index in the training data set, i=1,...,N; Represents the original image corresponding to the i-th image block; represents the initial reconstructed image corresponding to the i-th image block; represents the Fourier convolution reconstructed image corresponding to the i-th image block; f w (·) is the filter function learned by pixel-level filter; Represents the filtering result of the Fourier convolution reconstructed image corresponding to the i-th image block.

[0074] In one application scenario of the present invention, a training set is constructed after building a network structure from a sampling module to a fusion module. A data set 91image that is consistent with multiple mainstream compressed sensing reconstruction algorithms based on deep learning is selected. 36,184 image blocks are cropped out of each training image with a step size of 57 pixels as a training set, and the image block size is 64×64. The test set includes commonly used standard data sets such as Set11 and BSD68. Using the Adam optimization algorithm, the number of training epochs is 200, the batch size is 32, the learning rate is 10-4, and the filter size is set to 17. Then, FFC-CFFNet outputs a weight size of 17×17=289 at the pixel position of each image.

[0075] The compressed sensing reconstruction method provided by the present invention is compared with the existing MH algorithm, ReconNet, CSNet, and ISTANet+. It is found that under the Set11 data set, the average PSNR of the method proposed in the present invention is improved by 2.13dB, 5.50dB, 0.58dB, and 1.46dB respectively at a sampling rate of 10%, 2.28dB, 6.67dB, 1.88dB, and 0.41dB respectively at a sampling rate of 20%, and 2.16dB, 7.97dB, 2.73dB, and 0.01dB respectively at a sampling rate of 30%. Under the BSD68 dataset, the average PSNR of the reconstruction algorithm proposed in the present invention is improved by 2.69dB, 3.04dB, 0.54dB, and 1.36dB respectively at a sampling rate of 10% compared with the MH algorithm, ReconNet, CSNet, and ISTANet+, and is improved by 2.61dB, 4.47dB, 0.08dB, and 0.56dB respectively at a sampling rate of 20%, and is improved by 2.50dB, 5.02dB, 0.28dB, and 0.94dB respectively at a sampling rate of 30%. In the Setl 1 dataset, the average SSIM of the method proposed in the present invention is improved by 0.0865, 0.2442, 0.0275, and 0.0651 respectively at a sampling rate of 10%, by 0.0504, 0.1901, 0.0213, and 0.0094 respectively at a sampling rate of 20%, and by 0.0437, 0.1415, 0.0588, and 0.0118 respectively at a sampling rate of 30%, compared with the MH algorithm, ReconNet, CSNet, and ISTANet+. Under the BSD68 data set, the average SSIM of the method proposed in the invention is improved by 0.1337, 0.1414, 0.0172, and 0.0895 respectively at a sampling rate of 10%, 0.1127, 0.1768, 0.0063, and 0.0178 respectively at a sampling rate of 20%, and 0.0910, 0.1344, 0.0352, and 0.0404 respectively at a sampling rate of 30% compared with the MH algorithm, ReconNet, CSNet, and ISTANet+. It can be seen that the average PSNR and average SSIM of the method provided by the present invention are the best in reconstruction quality under the data set, indicating that the reconstruction system provided by the present invention is an image compression sensing reconstruction network model that can achieve high-quality reconstruction. PSNR refers to Peak Signal to Noise Ratio, and SSIM refers to Structure Similarity.

[0076] In this application scenario, the pixel-level filters learned by the convolutional filter flow are visualized and analyzed, such as Fig. 9As shown, the reconstructed image of the system of the present invention and the visualization of the corresponding filter flow at a sampling rate of 30% are shown. The model training learns the filter flow at the pixel level. The final reconstructed image is obtained by multiplying the reconstructed image of the fast Fourier convolution with the filter flow in different dimensions through the broadcast mechanism. 2Dt-SNE is used to assign a color to each centroid, and the nearest centroid is visualized at the position of each filter. The distribution of the expected filter flow vector is visualized by calculating the expected filter flow vector, showing that the different colors in the filter flow visualization are the changes in the corresponding spatial positions of the corresponding pixels, and the depth of the color indicates the intensity of the change. The visualization result provides an explicit statement, indicating what the convolution filter flow framework has learned. At the same time, the design of each module in the convolution filter flow model network architecture plays a role, whether it is the interpretability of the model architecture or the explicit statement provided, which is a significant advantage in interpretability compared to other convolutional neural network reconstruction models.

[0077] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0078] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. An image compression sensing method based on fast Fourier convolution and convolution filter flow, characterized in that: include: Obtaining multiple image patch measurements; Performing initial reconstruction on the measurement values ​​of the multiple image blocks to obtain an initial reconstructed image; The initial reconstructed image is further reconstructed based on the fast Fourier convolution to obtain a Fourier convolution reconstructed image; Input the Fourier convolution reconstructed image into a pixel-level filter based on a convolution filter flow network to obtain a filtering result, and convolve the filtering result with the Fourier convolution reconstructed image to obtain a final reconstructed image; The pixel-level filter includes a first convolutional neural network, a second convolutional neural network, a third convolutional neural network and a mapping layer; The Fourier convolution reconstructed image is input into the first convolutional neural network and the second convolutional neural network to obtain a first image and a second image respectively; Performing element-wise summation on the first image and the second image and sequentially inputting the summation results into the third convolutional neural network and the mapping layer for processing to obtain a filtering processing result; The depth of the first convolutional neural network is greater than the depth of the second convolutional neural network, and the third convolutional neural network includes multiple layers of convolution.

2. The image compression sensing method based on fast Fourier convolution and convolution filter flow according to claim 1, characterized in that: The step of obtaining the measurement values ​​of the multiple image blocks is specifically as follows: The original image is input into a sampling network to obtain a plurality of image patch measurement values, wherein the sampling network includes a sampling convolutional layer.

3. The image compression sensing method based on fast Fourier convolution and convolution filter flow according to claim 1, characterized in that: The step of initially reconstructing the measurement values ​​of the multiple image blocks to obtain the initial reconstructed image specifically includes: Input multiple image block measurement values ​​into the initial reconstruction convolution layer to obtain a vector representation of the image block; The vector representations of multiple image blocks are reshaped into an initial reconstructed image with the same size as the original image through a pixel shuffling operation.

4. The image compression sensing method based on fast Fourier convolution and convolution filter flow according to claim 1, characterized in that: The step of further reconstructing the initial reconstructed image based on fast Fourier convolution to obtain a Fourier convolution reconstructed image comprises: The initial reconstructed image is sequentially input into a fast Fourier convolution network, multiple fast Fourier convolution residual networks and the first convolution layer to obtain a global feature image; The initial reconstructed image and the global feature image are element-wise summed to obtain the Fourier convolution reconstructed image.

5. Image compression sensing system based on fast Fourier convolution and convolution filter flow, characterized in that: include: A sampling module, used for compressing and sampling the original image to obtain a plurality of image block measurement values; An initial reconstruction module, used for performing initial reconstruction on a plurality of image block measurement values ​​to obtain an initial reconstructed image; A Fourier convolution reconstruction module further reconstructs the initial reconstructed image based on fast Fourier convolution to obtain a Fourier convolution reconstructed image; A pixel-level filter is used to filter the Fourier convolution reconstructed image, wherein the pixel-level filter adopts a convolution filter flow network architecture; The fusion module convolves the filtering processing result of the pixel-level filter with the Fourier convolution reconstruction image to obtain the final reconstructed image; The pixel-level filter includes a first convolutional neural network, a second convolutional neural network, a third convolutional neural network, a second summation unit and a mapping layer, the depth of the first convolutional neural network is greater than the depth of the second convolutional neural network, and the third convolutional neural network includes multiple layers of convolution; The output end of the Fourier convolution reconstruction module is connected to the input end of the first convolutional neural network and the input end of the second convolutional neural network respectively, the output end of the first convolutional neural network and the output end of the second convolutional neural network are both connected to the input end of the second summation unit, the output end of the second summation unit is connected to the input end of the third convolutional neural network, and the output end of the third convolutional neural network is connected to the input end of the mapping layer.

6. The image compression sensing system based on fast Fourier convolution and convolution filter flow according to claim 5, characterized in that: The sampling module includes a sampling convolution layer; And / or, the initial reconstruction module includes an initial reconstruction convolutional layer and a pixel shuffling unit connected in sequence.

7. The image compression sensing system based on fast Fourier convolution and convolution filter flow according to claim 5, characterized in that: The Fourier convolution reconstruction module includes a first summing unit, and a fast Fourier convolution network, a plurality of fast Fourier convolution residual networks and a first convolution layer connected in sequence; The input end of the fast Fourier convolution network is connected to the output end of the initial reconstruction module, the output end of the first convolution layer is connected to the first input end of the first summing unit, the output end of the initial reconstruction module is also connected to the second input end of the first summing unit, and the first summing unit outputs the Fourier convolution reconstructed image.

8. The image compression sensing system based on fast Fourier convolution and convolution filter flow as claimed in claim 5, characterized in that: The loss function in the training of the image compression sensing system is: Wherein, N represents the total number of image blocks in the training data set; i represents the image block index in the training data set, i=1,...,N; Represents the original image corresponding to the i-th image block; represents the initial reconstructed image corresponding to the i-th image block; represents the Fourier convolution reconstructed image corresponding to the i-th image block; f w (·) is the filter function learned by the pixel-level filter; Represents the filtering result of the Fourier convolution reconstructed image corresponding to the i-th image block.

Images