Parallel convolution single-pixel detector imaging method based on physical information hybrid constraint

Abstract

The invention belongs to the technical field of computational imaging, and particularly relates to a physical information hybrid constraint-based parallel convolution single-pixel detector imaging method, which comprises the following steps of: S1, prefabricating a coding matrix by using a Hadamard matrix as a two-dimensional structured modulation light field; s2, obtaining one-dimensional light intensity coding information of a target scene through the output voltage of the single-pixel detector, carrying out parallel matrix, and establishing a parallel convolutional neural network model; s3, constructing a physical information constraint equation used for optimizing convolution parameters of the parallel convolutional neural network model; and S4, determining a physical information hybrid constraint condition, optimizing a convolution kernel of the parallel convolutional neural network model, and obtaining a target scene image. According to the method, neural network model parameters can be minimized in the imaging process of the single-pixel detector, the number of iterations is reduced, the imaging time is shortened, real-time single-pixel imaging under physical information driving is achieved, and the single-pixel detector can have higher imaging quality at a low sampling rate by introducing physical mixed constraints.

Description

Technical Field

[0001] The present invention belongs to the technical field of computational imaging, and in particular relates to a parallel convolution single-pixel detector imaging method based on physical information mixing constraints. Background Art

[0002] Optical imaging technology is the basis for humans to understand and perceive the natural world from a visual perspective. Traditional imaging methods rely on high-pixel array sensors, while single-pixel imaging technology can obtain scene spatial information using only a single pixel detector by modulating the light field. It has great advantages in wide spectrum, low bandwidth, high information dimension and specific application scenarios. However, its special imaging mechanism makes the imaging quality and the number of measurements mutually restrictive, which hinders the practical application of single-pixel imaging. How to use post-algorithm design to restore higher quality reconstructed images with fewer measurements has become the focus of many studies.

[0003] Due to the powerful learning ability of neural networks in the computational domain, applying deep learning technology to single-pixel imaging can effectively solve the balance problem. Many researchers have proposed data-driven and physical-driven neural network single-pixel imaging methods, achieving high-quality reconstruction at low sampling rates. However, the traditional data-driven neural network single-pixel imaging method has the problem of poor performance in unknown scenes, as well as generalization problems and unexplainability in the case of incomplete data sets. The new non-training physical-driven neural network single-pixel imaging method is interpretable, but it has the problem of low efficiency and long imaging time. Therefore, on the basis of ensuring generalization and interpretability, there is still room for improvement in imaging efficiency. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a parallel convolution single-pixel detector imaging method based on physical information mixing constraints. During the single-pixel detector imaging process, the neural network model parameters can be minimized, the number of iterations can be reduced, the imaging time can be shortened, and real-time single-pixel imaging driven by physical information can be realized. The introduction of physical mixing constraints can enable the single-pixel detector to have higher imaging quality at a lower sampling rate.

[0005] To achieve the above object, the present invention discloses the following technical solution:

[0006] A parallel convolution single-pixel detector imaging method based on physical information mixing constraints, comprising:

[0007] S1: Use the Hadamard basis matrix to preform the coding matrix as a two-dimensional structured modulated light field;

[0008] Determine the reconstructed image resolution to be N×N, fix the sampling rate to M / N×N, and select N 2 ×N 2The M rows of the Hadamard basis matrix generate the two-dimensional structured coding matrix P i (x, y), i = 1, 2, 3, ..., M, is used as the modulated light field at the sampling rate for target scene encoding;

[0009] S2: Obtain the one-dimensional light intensity encoding information of the target scene through the output voltage of the single-pixel detector, perform parallel matrixing, and establish a parallel convolutional neural network model;

[0010] S21: The two-dimensional structured modulated light field generated in step S1 interacts with the target object to be measured, and a coding detection matrix [S1, S2, ..., S M ] T ;

[0011] S22: The data acquisition time per second is set to t1, and the algorithm reconstruction time is set to t2, where t1+t2=1s; the flip frequency of the fixed digital micromirror device is set to v, and the real-time imaging frame rate F=t1×v / M is obtained, so that the single-pixel detector detects F×M light intensity signal values, and the single-pixel detection value is parallel matrixed as follows:

[0012]

[0013] Among them, S is the parallel matrix of single pixel detection values; S ki is the i-th single pixel detection value of the imaging frame rate at time k; M is the fixed sampling number; F is the real-time imaging frame rate;

[0014] S23: Fitting the physical process of single-pixel imaging, setting the target scene to be measured as the mapping function f, and using the two-dimensional structured coding matrix P in step S1 i (x, y) is used as input to establish a parallel convolutional neural network model:

[0015]

[0016] Among them, S' is the parallel matrix of network output simulation values; f is the real physical model mapping function; f θ is the physical fitting mapping function of the neural network; θ is the neural network parameter set;

[0017] The parallel convolutional neural network model uses F parallel convolutional layers with the same initialization parameters as the imaging resolution size at the real-time imaging frame rate as the network intermediate layer, fits the physical process of single-pixel imaging through the convolution process of the network input and the intermediate parallel convolutional layer, and obtains the parallel matrix S' of the network output simulation value;

[0018] S3: constructing a physical information constraint equation for optimizing the convolution parameters of the parallel convolutional neural network model in step S2; the physical information constraint equation includes: a physical information detection value constraint, a total variation regularization constraint, a sparse constraint, and a non-negative constraint;

[0019] S4: Determine the physical information mixing constraint conditions, optimize the convolution kernel of the parallel convolutional neural network model, and obtain the target scene image;

[0020] According to the physical information detection value loss function Loss1, the total variation regularization constraint loss function Loss2, the sparse constraint loss function Loss3 and the non-negative constraint in step S3, the physical information mixed constraint condition is formed, and the neural network parameter set θ is updated as:

[0021]

[0022] Among them, J(θ) is the overall loss function of physical information mixed constraint; Loss1 is the physical information detection value constraint loss function; Loss2 is the total variation regularization constraint loss function; Loss3 is the sparse constraint loss function; λ is the weight parameter of the total variation regularization weight parameter; μ is the weight parameter of the sparse prior penalty term; is the gradient of the loss function relative to the parameter; l is the network learning rate;

[0023] The Adam optimizer is used to iteratively optimize the convolution layer parameters θ of the parallel convolutional neural network model, and the hyperparameters are adjusted: the weight parameters λ, μ of the mixed constraint and the network learning rate l, so that the convolution kernel parameters of the parallel convolutional neural network model are close to the real image of the real parallelization, so that the single-pixel detector can complete the imaging of the target scene.

[0024] Preferably, a two-dimensional structured coding matrix is ​​generated in step S1, specifically: the negative terms in the Hadamard basis matrix are set to zero to comply with the non-negative constraint in the light field, and the number of connected areas contained in each row of the Hadamard basis matrix is ​​arranged from small to large, and the first M rows are taken as the coding matrix at the sampling rate, and the coding matrix is ​​proportionally enlarged into the micromirror according to the resolution of the digital micromirror device as a two-dimensional structured coding matrix.

[0025] Preferably, in step S21, a one-dimensional coded detection value is formed according to the obtained single-pixel detection value to form a coded detection matrix, specifically:

[0026] The two-dimensional structured modulated light field generated in step S1 interacts with the target object to be measured to obtain a single-pixel detection value, and the single-pixel imaging forward physical signal acquisition process is realized. After M times of modulated light field encoding, a one-dimensional coded detection value S is obtained. i (i=1,2,3,…,M), specifically:

[0027]

[0028] Among them, S i is the i-th one-dimensional coded detection value; i is the one-dimensional coded detection value number; O(x, y) is the two-dimensional light intensity matrix of the cross section of the object to be measured; o N is the Nth element of the cross-section matrix of the object to be measured; P i (x, y) is a two-dimensional structured coding matrix generated based on the Hadamard basis matrix; p iN is the Nth element of the i-th two-dimensional structured coding matrix generated based on the Hadamard basis matrix; (x, y) is the Cartesian coordinate of the two-dimensional interaction surface between the target scene to be measured and the two-dimensional structured light field in space; x is the abscissa of the two-dimensional interaction surface between the target scene to be measured and the two-dimensional structured light field; y is the ordinate of the two-dimensional interaction surface between the target scene to be measured and the two-dimensional structured light field;

[0029] The coded detection matrix is ​​constructed based on the one-dimensional coded detection values ​​measured M times:

[0030]

[0031] Among them, p ij is the j-th element of the i-th two-dimensional structured coding matrix generated based on the Hadamard basis matrix.

[0032] Preferably, in step S22, single pixel detection values ​​are parallelized into matrices, specifically:

[0033] Since the negative terms in the Hadamard basis cannot be directly converted on the digital micromirror device, positive and negative projected structured light fields are required, and the detector values ​​are differentially calculated to ensure orthogonality; the negative projected light field of the detector is:

[0034] P - =P 白 -P +

[0035] Among them, P - The projected light field with the absolute value of the negative item corresponding to the encoding matrix and the positive item set to 0; P 白 is the all-white projection light field; P + The positive items corresponding to the encoding matrix are retained, and the negative items are set to 0 for the projected light field;

[0036] The differential detection value of the detector is:

[0037] S + -S - =P + O(x,y)-P - O(x,y)=2P + O(x,y)-P 白O(x,y)

[0038] Among them, S + is the light intensity detection value corresponding to the forward projection light field; S - is the light intensity detection value corresponding to the negative projection light field; O(x, y) is the two-dimensional light intensity matrix of the cross section of the object to be measured that interacts with the projection light field;

[0039] The differential detection value is calculated from the positive projection light field and the full white light field, without the need to project the negative projection light field, and the actual sampling number is equal to the simulated sampling number.

[0040] Preferably, the initialization parameters in step S23 are the same as the physical information of single-pixel imaging fitted by the parallel convolutional neural network, and a batch of coding matrices of size N×N, i.e., digitized modulated light fields, are required as the input of the parallel convolutional neural network, and F parallel convolutional layers of the same size as the input imaging resolution are passed to output parallel detection values ​​of size F×M; the parameters of the convolutional layer of the parallel convolutional neural network are iteratively optimized through the output and the known single-pixel detector values, so the parameters of the network convolutional layer are initialized and not trained.

[0041] Preferably, the parallel convolution layer in step S23 specifically parallelizes the convolution layer in a single convolution kernel neural network that fits the physical information reconstructed by a single image, uses z convolution kernels to realize parallel image processing when the input is unified, each convolution kernel calculates its own output, and then outputs them in parallel to calculate the loss function.

[0042] Preferably, the physical information detection value constraint, total variation regularization constraint, sparse constraint and non-negative constraint in step S3 are specifically:

[0043] S31: The network output simulation value parallel matrix S' and the single pixel detection value parallel matrix S in step S23 are used as data fidelity items in the physical information mixed loss function to obtain the physical information detection value constraint loss function Loss1:

[0044]

[0045] Among them, Loss1 is the physical information detection value constraint loss function; S' is the network output simulation value parallel matrix; S is the single pixel detection value parallel matrix;

[0046] S32: The middle layer of the parallel convolutional neural network model fits the target object to be measured in the physical process, and establishes the total variation regularization constraint loss function Loss2 as:

[0047]

[0048] Among them, Loss2 is the total variation regularization constraint loss function; θ x,yis the parameter value of the convolution kernel at (x, y);

[0049] S33: The sparse constraint loss function Loss3 is determined by the LO norm constraint:

[0050] Loss3=||θ x,y ||0

[0051] Among them, Loss3 is the sparse constraint loss function;

[0052] S34: The non-negative prior is used as the final constraint condition update method: θ x,y ←max(θ x,y ,0).

[0053] Preferably, the element values ​​in the network output simulation value parallel matrix S' and the single pixel detection value parallel matrix S in step S31 need to be normalized with zero mean and unit variance, specifically:

[0054]

[0055] in, is the result of row zero mean and unit variance standardization; m k is the mean value of the kth row in the parallel matrixed detection value, that is, the mean value of the light intensity detection value in the kth frame; s k is the standard deviation of the kth row in the parallel matrixed detection values, that is, the standard deviation of the light intensity detection values ​​in the kth frame.

[0056] Preferably, in step S4, the physical information mixed constraint condition is used as the final loss function together with the physical information detection value loss constraint through total variation regularization and sparsity constraint, and the non-negative constraint acts solely on the optimized convolution kernel parameters, so that the negative terms in the convolution kernel parameters are set to 0 to ensure the non-negative constraint while further increasing the sparsity.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] (1) The present invention proposes a method for realizing imaging using convolution kernel weights by fitting the forward physical information of single-pixel imaging with a parallel convolutional neural network, thereby realizing an efficient reconstruction algorithm with minimal parameters. The input of the network is a coding matrix corresponding to the modulated light field, and the label required for loss calculation is a true single-pixel detection value. Therefore, the present invention does not require data set pre-training. At the same time, since the method is calculated in a GPU, it can simultaneously ensure imaging quality and reconstruction time in actual use to achieve fast imaging under the premise of high-quality reconstruction.

[0059] (2) The present invention processes multiple frames of images simultaneously through parallel convolutional layers, effectively avoiding the problem of long iteration time for each imaging in single-pixel imaging of physical information; the parallel convolutional network is flexible, and the network structure does not need to be changed when replacing different types of encoding matrices and realizing imaging of different resolutions; the additional physical prior mixed constraints reduce the range of parameter changes, improve accuracy, and enable the network to converge quickly, and the comprehensive data acquisition time can achieve video frame ≥ 24 frame imaging. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a flow chart of the parallel convolution single pixel detector imaging method based on physical information mixing constraint of the present invention;

[0061] Figure 2 A schematic diagram of the Hadamard basis matrix used in the experimental process of the present invention;

[0062] Figure 3 A schematic diagram of a coding matrix generated based on the Hadamard basis during the experimental process of the present invention;

[0063] Figure 4 It is a schematic diagram of the standardized light intensity detection value of the present invention;

[0064] Figure 5 It is a schematic diagram of the calculation of the frame rate of the stage acquisition of the present invention;

[0065] Figure 6 It is a schematic diagram of continuous acquisition frame rate calculation of the present invention;

[0066] Figure 7 The structure and constraint diagram of the parallel convolutional network with physical information hybrid constraints of the present invention;

[0067] Figure 8 This is an imaging diagram of a rotating object at a sampling rate of 10% according to the present invention;

[0068] Fig. 9 This is an imaging diagram of a translation object at a sampling rate of 10% according to the present invention. DETAILED DESCRIPTION

[0069] The exemplary embodiments, features and aspects of the present invention will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0070] The embodiment of the present invention provides a parallel convolution single pixel detector imaging method based on physical information mixing constraints, such as Figure 1As shown, a Hadamard matrix is ​​used to prefabricate a coding matrix as a two-dimensional structured modulated light field; the one-dimensional light intensity coding information of the target scene is obtained through the output voltage of a single-pixel detector, and parallel matrixing is performed to establish a parallel convolutional neural network model; a physical information constraint equation for optimizing the convolution parameters of the parallel convolutional neural network model is constructed; the physical information mixing constraint conditions are determined, the convolution kernel of the parallel convolutional neural network model is optimized, and the target scene image is obtained; it includes:

[0071] Step S1: Use the Hadamard basis matrix to preform a coding matrix as a two-dimensional structured modulated light field.

[0072] Determine the reconstructed image resolution to be N×N, fix the sampling rate to M / N×N, and select N 2 ×N 2 The M rows of the Hadamard basis matrix generate the two-dimensional structured coding matrix P i (x, y), i = 1, 2, 3, ..., M, specifically: the negative terms in the Hadamard basis matrix are set to zero to meet the non-negative constraints in the light field, and the number of connected areas contained in each row of the Hadamard basis matrix is ​​arranged from small to large, and the first M rows are taken as the coding matrix at the sampling rate. According to the resolution of the digital micromirror device, the coding matrix is ​​proportionally enlarged into the micromirror as a two-dimensional structured coding matrix, which is used for the modulated light field at the sampling rate for target scene encoding.

[0073] The embodiment of the present invention is to reconstruct a grayscale image with a resolution of 64 pixels × 64 pixels. The number of full sampling is 4096, and the number of undersampling is 409. The sampling rate is about 10%. Figure 2 The figure shows a schematic diagram of the Hadamard basis matrix used in the experimental process of the present invention; a two-dimensional structured coding matrix is ​​generated based on the Hadamard basis matrix with a dimension of 4096, and the negative items in the Hadamard matrix are set to zero to meet the non-negative constraint in the light field. At the same time, the number of connected areas contained in each row of the Hadamard matrix is ​​arranged from small to large, and the first 409 rows are taken as the coding matrix under the sampling rate, such as Figure 3 Shown is a schematic diagram of a coding matrix generated based on the Hadamard basis during the experimental process of the present invention.

[0074] According to the resolution of the digital micromirror device, the coding matrix is ​​proportionally magnified into the micromirror as a two-dimensional structured light field. In the embodiment of the present invention, the pixel size of the digital micromirror device is 1024pixel×768pixel, so the magnification ratio is 12 times, the 64pixel×64pixel coding matrix is ​​magnified to 768pixel×768pixel, and the remaining horizontal pixels are set to 0 to form a two-dimensional structured light field with the same pixel size as the digital micromirror device, and the light field image pre-made by the computer is saved as a .bmp file and loaded into the digital micromirror device for subsequent scene interaction.

[0075] Step S2: Obtain the one-dimensional light intensity encoding information of the target scene through the output voltage of the single-pixel detector, perform parallel matrixing, and establish a parallel convolutional neural network model.

[0076] Step S21: The two-dimensional structured modulated light field generated in step S1 interacts with the target object to be measured to obtain a single-pixel detection value, realize the single-pixel imaging forward physical signal acquisition process, and obtain a one-dimensional coded detection value S after M times of modulated light field coding. i (i=1,2,3,…,M), specifically:

[0077]

[0078] Among them, S i is the i-th one-dimensional coded detection value; i is the one-dimensional coded detection value number; O(x, y) is the two-dimensional light intensity matrix of the cross section of the object to be measured; o N is the Nth element of the cross-section matrix of the object to be measured; P i (x, y) is a two-dimensional structured coding matrix generated based on the Hadamard basis matrix; p iN is the Nth element of the i-th two-dimensional structured coding matrix generated based on the Hadamard basis matrix; (x, y) is the Cartesian coordinate of the two-dimensional interaction surface between the target scene to be measured and the two-dimensional structured light field in space; x is the abscissa of the two-dimensional interaction surface between the target scene to be measured and the two-dimensional structured light field; y is the ordinate of the two-dimensional interaction surface between the target scene to be measured and the two-dimensional structured light field.

[0079] The coded detection matrix is ​​constructed based on the one-dimensional coded detection values ​​measured M times:

[0080]

[0081] Among them, p ij is the j-th element of the i-th two-dimensional structured coding matrix generated based on the Hadamard basis matrix.

[0082] like Figure 4 The figure shows a schematic diagram of the standardized light intensity detection value of the present invention, which is a visualized image of a one-dimensional time series of the standardized light intensity detection signal. The experimental process of the embodiment of the present invention is as follows: the scene to be tested is focused on the effective pixel range of the digital micromirror device through a lens, so that the scene to be tested interacts with the modulated light field, and different modulated light fields are formed through the ±12° deflection of the micromirror, and a focusing mirror is placed on the flip surface to focus the total light intensity value of the scene into a single pixel detector.

[0083] Step S22: The data acquisition time per second is set to t1, and the algorithm reconstruction time is set to t2, where t1+t2=1s; the fixed digital micromirror device flip frequency is set to v, and the real-time imaging frame rate F=t1×v / M is obtained, so that the single pixel detector detects F×M light intensity signal values, specifically:

[0084] S=[S 11 ,S 12 ,S 13 ,…,S 1M ,S 21 ,S 22 ,…,S FM ] T .

[0085] The one-dimensional single pixel detection values ​​of the above F frames are parallelized into:

[0086]

[0087] Among them, S is the parallel matrix of single pixel detection values; S ki is the i-th single pixel detection value of the imaging frame rate at time k; M is the fixed sampling number; F is the real-time imaging frame rate.

[0088] In the embodiment of the present invention, the modulation frequency of the digital micromirror device is set to 22KHz, the sampling times is 409, and the time required for the subsequent algorithm reconstruction link is 0.29s. The maximum sampling time within one second is 0.71s, and the real-time reconstruction frame rate is expressed as:

[0089] F = (0.71 × 22000) / 409 ≈ 38;

[0090] Take integers and perform parallel matrixing to get S F×M .

[0091] like Figure 5 The figure shows the stage acquisition frame rate calculation diagram of the present invention; sampling stops during reconstruction to ensure a fixed frame rate of 38 per second. If sampling is not stopped during reconstruction, the reconstruction frame rate is maximized; Figure 6 The figure shows a schematic diagram of the continuous acquisition frame rate calculation of the present invention; at the first second, the number of reconstructed image frames is still 38, while at the second second and thereafter, 38 frames of images can be reconstructed every 0.71 seconds, and the reconstruction frame rate F max =38 / 0.71≈53 frames.

[0092] Perform parallel matrixing of single pixel detection values, specifically:

[0093] Since the negative terms in the Hadamard basis cannot be directly converted on the digital micromirror device, positive and negative projected structured light fields are required, and the detector values ​​are differentially calculated to ensure orthogonality; the negative projected light field of the detector is:

[0094] P - =P 白 -P + ;

[0095] Among them, P - The projected light field with the absolute value of the negative item corresponding to the encoding matrix and the positive item set to 0; P 白 is the all-white projection light field; P + The positive items corresponding to the encoding matrix are retained and the negative items are set to 0 in the projected light field.

[0096] The differential detection value of the detector is:

[0097] S + -S - =P + O(x,y)-P - O(x,y)=2P + O(x,y)-P 白 O(x,y);

[0098] Among them, S + is the light intensity detection value corresponding to the forward projection light field; S - is the light intensity detection value corresponding to the negative projection light field; O(x, y) is the two-dimensional light intensity matrix of the cross section of the object to be measured that interacts with the projection light field.

[0099] The differential detection value can be calculated from the positive projection light field and the full white light field without projecting the negative projection light field, and the actual sampling number is equal to the simulated sampling number.

[0100] Step S23: Fitting the physical process of single-pixel imaging, setting the target scene to be measured as the mapping function f, and using the two-dimensional structured coding matrix P in step S1 i (x, y) is used as input to establish a parallel convolutional neural network model:

[0101]

[0102] Among them, S' is the parallel matrix of network output simulation values; f is the real physical model mapping function; f θ is the physical fitting mapping function of the neural network; θ is the set of neural network parameters.

[0103] The parallel convolutional neural network model uses F parallel convolutional layers with the same initialization parameters as the imaging resolution size at the real-time imaging frame rate as the network intermediate layer. The physical process of single-pixel imaging is fitted through the convolution process of the network input and the intermediate parallel convolutional layer to obtain the parallel matrix S' of the network output simulation value.

[0104] The initialization parameters are the same as those of the parallel convolutional neural network for fitting single-pixel imaging physical information. A batch of encoding matrices of size N×N, namely the digital modulated light field, are required as the input of the parallel convolutional neural network. Through F parallel convolutional layers of the same size as the input imaging resolution, parallel detection values ​​of size F×M are output. The parameters of the convolutional layer of the parallel convolutional neural network are iteratively optimized through the output and the known single-pixel detector values, so the parameters of the network convolutional layer are initialized and not trained.

[0105] The parallel convolution layer specifically parallelizes the convolution layer in a single convolution kernel neural network that fits the physical information reconstructed from a single image, uses z convolution kernels to process images in parallel with unified input, and each convolution kernel calculates its own output, which is then output in parallel to calculate the loss function.

[0106] The physical process of single-pixel imaging is to illuminate the target object with a modulated light field, and the overall light intensity value of the scene is collected by a single-pixel detector. In this process, the target object is mapped from the modulated light field to a series of one-dimensional measurement values, which are fitted into the neural network, and the image is reconstructed as an intermediate process.

[0107] The neural network model is designed based on the forward physical information mentioned above. The designed parallel convolutional neural network model uses the two-dimensional structured coding matrix in step S1 as the network input, and uses F parallel convolution kernels with the same initialization parameters as the imaging resolution size as the network intermediate layer. The physical process of single-pixel imaging is fitted through the convolution process of the network input and the intermediate parallel convolution kernel to obtain the parallel matrixed simulated detection value. The parallel convolution layer specifically parallelizes the convolution layer in the single convolution kernel neural network that fits the physical information of single image reconstruction, and uses z convolution kernels to realize parallel processing of multiple or multiple frames of images when unified input is used. Each convolution kernel calculates its own output, and then outputs the output in parallel to calculate the loss uniformly. In an embodiment of the present invention, the network input dimension is (409, 1, 64, 64), that is, corresponding to 409 modulated light fields, its effective pixels are 64 pixels × 64 pixels, and 38 parallel convolution layers are passed through in the middle, and the output dimension is modified to (38, 409) corresponding to the real detection value.

[0108] Step S3: construct a physical information constraint equation for optimizing the convolution parameters of the parallel convolutional neural network model in step S2; the physical information constraint equation includes: physical information detection value constraint, total variation regularization constraint, sparsity constraint and non-negative constraint.

[0109] Step S31: The network output simulation value parallel matrix S' and the single pixel detection value parallel matrix S in step S23 are used as data fidelity items in the physical information mixed loss function to obtain the physical information detection value constraint loss function Loss1:

[0110]

[0111] Among them, Loss1 is the physical information detection value constraint loss function; S' is the network output simulation value parallel matrix; S is the single pixel detection value parallel matrix.

[0112] The element values ​​in the network output simulation value parallel matrix S' and the single pixel detection value parallel matrix S need to be normalized with zero mean and unit variance, specifically:

[0113]

[0114] in, is the result of row zero mean and unit variance standardization; m k is the mean value of the kth row in the parallel matrixed detection value, that is, the mean value of the light intensity detection value in the kth frame; s k is the standard deviation of the kth row in the parallel matrixed detection values, that is, the standard deviation of the light intensity detection values ​​in the kth frame.

[0115] Step S32: The intermediate layer of the parallel convolutional neural network model fits the target object to be measured in the physical process, and establishes the total variation regularization constraint loss function Loss2 as:

[0116]

[0117] Among them, Loss2 is the total variation regularization constraint loss function; θ x,y is the parameter value of the convolution kernel at (x, y).

[0118] Step S33: Determine the sparse constraint loss function Loss3 by the LO norm constraint:

[0119] Loss3=||θ x,y ||0;

[0120] Among them, Loss3 is the sparse constraint loss function.

[0121] Step S34: The non-negative prior is used as the final constraint and does not participate in the loss calculation. The parameters are directly updated as follows:

[0122] θ x,y ←max(θ x,y ,0).

[0123] Step S4: Determine the physical information mixing constraint conditions, optimize the convolution kernel of the parallel convolutional neural network model, and obtain the target scene image.

[0124] According to the physical information detection value loss function Loss1, the total variation regularization constraint loss function Loss2, the sparse constraint loss function Loss3 and the non-negative constraint in step S3, the physical information mixed constraint condition is:

[0125]

[0126] The iterative update parameters are expressed as:

[0127]

[0128] Among them, J(θ) is the overall loss function of the physical information mixed constraint; λ is the weight parameter of the total variation regularization weight parameter; μ is the weight parameter of the sparse prior penalty term; is the gradient of the loss function with respect to the parameters; l is the network learning rate.

[0129] The physical information mixed constraint condition is used as the final loss function together with the physical information detection value loss constraint through total variation regularization and sparsity constraint. The non-negative constraint acts solely on the optimized convolution kernel parameters, setting the negative terms in the convolution kernel parameters to 0 to ensure the non-negative constraint while further increasing sparsity.

[0130] The Adam optimizer is used to iteratively optimize the convolution layer parameters θ of the parallel convolutional neural network model, and the hyperparameters are adjusted at the same time: the weight parameters λ, μ of the mixed constraints and the network learning rate l, so that the convolution kernel parameters of the parallel convolutional neural network model are close to the truly parallelized multi-frame real images to obtain single-pixel detector imaging images.

[0131] like Figure 7 The parallel convolution network structure and constraint diagram of the physical information hybrid constraint of the present invention are shown, which is the physical information fitting parallel convolution neural network framework of the present invention, and the convolution kernel parameters are subject to multiple physical loss constraints. In the instantiation of the present invention, the network is iterated using the Adam optimizer, the hybrid constraint weight parameters l, m are 0.2 and 0.1 respectively, the network initial learning rate is set to 0.01, and the StepLR mechanism is set to adjust the learning rate, and the number of iterations is set to 200. After continuous iteration, the convolution kernel parameters are close to the real parallelized multi-frame real image. Finally, z parallel convolution kernel parameters are read as the multi-frame image of the final output.

[0132] like Figure 8 The figure shows the imaging of a rotating object at a sampling rate of 10% of the present invention; the imaging results of the rotating object from 1 to 180 frames at a sampling rate of 10%, and the image resolution is 64 pixels × 64 pixels. Thanks to the physical mixing constraint, the algorithm reconstruction time is 0.29 seconds after 200 iterations, and the average structural similarity performance of the imaging results of different frames reaches 96%. Fig. 9The image of the present invention at a sampling rate of 10% for a translation object is shown. The imaging results of the lower part of the frames 1 to 90 of the left-right translation object at a sampling rate of 10% are shown. The image resolution is 64 pixels × 64 pixels. In the first frame, the image structure similarity is 96.8%, and the peak signal-to-noise ratio is 28.5. The video frame imaging is achieved while also ensuring the imaging quality.

[0133] The beneficial effects of the present invention are as follows: the present invention provides a parallel convolution single-pixel detector imaging method based on physical information hybrid constraints. In view of the problem that the single-pixel imaging method based on physical information rather than training neural networks can effectively reconstruct unknown scene images but the reconstruction time is slow, a method for realizing imaging using convolution kernel weights by fitting the forward physical information of single-pixel imaging through parallel convolution neural networks that are first collected and then processed is proposed, thereby realizing an efficient reconstruction algorithm with few parameters; wherein the input of the network is the coding matrix corresponding to the modulated light field, and the label required for the loss calculation is the real single-pixel detection value, so the present invention does not require data set pre-training; at the same time, since the method is operated in a GPU, it can simultaneously ensure the imaging quality and reconstruction time in actual use to achieve fast imaging under the premise of high-quality reconstruction. The parallel convolution layer processes multiple frames of images at the same time, effectively avoiding the problem of long imaging iteration time for each imaging in physical information single-pixel imaging; at the same time, the network structure is simple, the hyperparameter adjustment is flexible, and the network structure does not need to be changed by replacing different types of coding matrices and realizing imaging of different resolutions; the additional physical priori hybrid constraints reduce the parameter variation range, improve the accuracy, and enable the network to converge quickly, which can not only realize imaging of video frames ≥ 24 frames, but also ensure that the imaging quality structure similarity is ≥ 95%.

[0134] The embodiments described above are only descriptions of the preferred implementation modes of the present invention, and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A parallel convolution single-pixel detector imaging method based on physical information mixing constraints, characterized in that: It includes: S1: Use the Hadamard basis matrix to preform the coding matrix as a two-dimensional structured modulated light field; Determine the reconstructed image resolution to be N×N, fix the sampling rate to M / N×N, and select N 2 ×N 2 The M rows of the Hadamard basis matrix generate the two-dimensional structured coding matrix P i (x, y), i = 1, 2, 3, ..., M, is used as the modulated light field at the sampling rate for target scene encoding; S2: Obtain the one-dimensional light intensity encoding information of the target scene through the output voltage of the single-pixel detector, perform parallel matrixing, and establish a parallel convolutional neural network model; S21: The two-dimensional structured modulated light field generated in step S1 interacts with the target object to be measured, and a coding detection matrix [S1, S2, ..., S M ] T ; S22: The data acquisition time per second is set to t1, and the algorithm reconstruction time is set to t2, where t1+t2=1s; the flip frequency of the fixed digital micromirror device is set to v, and the real-time imaging frame rate F=t1×v / M is obtained, so that the single-pixel detector detects F×M light intensity signal values, and the single-pixel detection value is parallel matrixed as follows: Among them, S is the parallel matrix of single pixel detection values; S ki is the i-th single pixel detection value of the imaging frame rate at time k; M is the fixed sampling number; F is the real-time imaging frame rate; S23: Fitting the physical process of single-pixel imaging, setting the target scene to be measured as the mapping function f, and using the two-dimensional structured coding matrix P in step S1 i (x, y) is used as input to establish a parallel convolutional neural network model: Among them, S' is the parallel matrix of network output simulation values; f is the real physical model mapping function; f θ is the physical fitting mapping function of the neural network; θ is the neural network parameter set; The parallel convolutional neural network model uses F parallel convolutional layers with the same initialization parameters as the imaging resolution size at the real-time imaging frame rate as the network intermediate layer, fits the physical process of single-pixel imaging through the convolution process of the network input and the intermediate parallel convolutional layer, and obtains the parallel matrix S' of the network output simulation value; S3: constructing a physical information constraint equation for optimizing the convolution parameters of the parallel convolutional neural network model in step S2; the physical information constraint equation includes: a physical information detection value constraint, a total variation regularization constraint, a sparse constraint, and a non-negative constraint; S4: Determine the physical information mixing constraint conditions, optimize the convolution kernel of the parallel convolutional neural network model, and obtain the target scene image; According to the physical information detection value loss function Loss1, the total variation regularization constraint loss function Loss2, the sparse constraint loss function Loss3 and the non-negative constraint in step S3, the physical information mixed constraint condition is formed, and the neural network parameter set θ is updated as: Among them, J(θ) is the overall loss function of physical information mixed constraint; Loss1 is the physical information detection value constraint loss function; Loss2 is the total variation regularization constraint loss function; Loss3 is the sparse constraint loss function; λ is the weight parameter of the total variation regularization weight parameter; μ is the weight parameter of the sparse prior penalty term; is the gradient of the loss function relative to the parameter; l is the network learning rate; The Adam optimizer is used to iteratively optimize the convolution layer parameters θ of the parallel convolutional neural network model, and the hyperparameters are adjusted: the weight parameters λ, μ of the mixed constraint and the network learning rate l, so that the convolution kernel parameters of the parallel convolutional neural network model are close to the real image of the real parallelization, so that the single-pixel detector can complete the imaging of the target scene.

2. The parallel convolution single-pixel detector imaging method based on physical information mixing constraints according to claim 1, characterized in that: In step S1, a two-dimensional structured coding matrix is ​​generated, specifically: the negative terms in the Hadamard basis matrix are set to zero to meet the non-negative constraints in the light field, and at the same time, the number of connected areas contained in each row of the Hadamard basis matrix is ​​arranged from small to large, and the first M rows are taken as the coding matrix at the sampling rate, and the coding matrix is ​​proportionally enlarged into the micromirror according to the resolution of the digital micromirror device as a two-dimensional structured coding matrix.

3. The parallel convolution single-pixel detector imaging method based on physical information mixing constraints according to claim 1, characterized in that: In step S21, a one-dimensional coded detection value is formed according to the obtained single pixel detection value to form a coded detection matrix, specifically: The two-dimensional structured modulated light field generated in step S1 interacts with the target object to be measured to obtain a single-pixel detection value, and the single-pixel imaging forward physical signal acquisition process is realized. After M times of modulated light field encoding, a one-dimensional coded detection value S is obtained. i (i=1,2,3,…,M), specifically: Among them, S i is the i-th one-dimensional coded detection value; i is the one-dimensional coded detection value number; O(x, y) is the two-dimensional light intensity matrix of the cross section of the object to be measured; o N is the Nth element of the cross-section matrix of the object to be measured; P i (x, y) is a two-dimensional structured coding matrix generated based on the Hadamard basis matrix; p iN is the Nth element of the i-th two-dimensional structured coding matrix generated based on the Hadamard basis matrix; (x, y) is the Cartesian coordinate of the two-dimensional interaction surface between the target scene to be measured and the two-dimensional structured light field in space; x is the abscissa of the two-dimensional interaction surface between the target scene to be measured and the two-dimensional structured light field; y is the ordinate of the two-dimensional interaction surface between the target scene to be measured and the two-dimensional structured light field; The coded detection matrix is ​​constructed based on the one-dimensional coded detection values ​​measured M times: Among them, p ij is the j-th element of the i-th two-dimensional structured coding matrix generated based on the Hadamard basis matrix.

4. The parallel convolution single-pixel detector imaging method based on physical information mixing constraints according to claim 1, characterized in that: In step S22, the single pixel detection value is parallelized into a matrix, specifically: Since the negative terms in the Hadamard basis cannot be directly converted on the digital micromirror device, positive and negative projected structured light fields are required, and the detector values ​​are differentially calculated to ensure orthogonality; the negative projected light field of the detector is: P - =P 白 -P + Among them, P - The projected light field with the absolute value of the negative item corresponding to the encoding matrix and the positive item set to 0; P 白 is the all-white projection light field; P + The positive items corresponding to the encoding matrix are retained, and the negative items are set to 0 for the projected light field; The differential detection value of the detector is: S + -S - =P + O(x,y)-P - O(x,y)=2P + O(x,y)-P 白 O(x,y) Among them, S + is the light intensity detection value corresponding to the forward projection light field; S - is the light intensity detection value corresponding to the negative projection light field; O(x, y) is the two-dimensional light intensity matrix of the cross section of the object to be measured that interacts with the projection light field; The differential detection value is calculated from the positive projection light field and the full white light field, without the need to project the negative projection light field, and the actual sampling number is equal to the simulated sampling number.

5. The parallel convolution single-pixel detector imaging method based on physical information mixing constraints according to claim 1, characterized in that: The initialization parameters in step S23 are the same as those of the parallel convolutional neural network for fitting single-pixel imaging physical information. A batch of encoding matrices of size N×N, i.e., digitized modulated light fields, are required as the input of the parallel convolutional neural network. Through F parallel convolutional layers of the same size as the input imaging resolution, parallel detection values ​​of size F×M are output. The parameters of the convolutional layer of the parallel convolutional neural network are iteratively optimized through the output and the known single-pixel detector values, so the parameters of the network convolutional layer are initialized and not trained.

6. The parallel convolution single pixel detector imaging method based on physical information mixing constraints according to claim 1, characterized in that: The parallel convolution layer in step S23 specifically parallelizes the convolution layer in a single convolution kernel neural network that fits the physical information reconstructed by a single image, uses z convolution kernels to realize parallel image processing when the input is unified, and each convolution kernel calculates its own output, which is then output in parallel to calculate the loss function.

7. The parallel convolution single-pixel detector imaging method based on physical information mixing constraints according to claim 1, characterized in that: The physical information detection value constraint, total variation regularization constraint, sparse constraint and non-negative constraint in step S3 are specifically: S31: The network output simulation value parallel matrix S' and the single pixel detection value parallel matrix S in step S23 are used as data fidelity items in the physical information mixed loss function to obtain the physical information detection value constraint loss function Loss1: Among them, Loss1 is the physical information detection value constraint loss function; S' is the network output simulation value parallel matrix; S is the single pixel detection value parallel matrix; S32: The middle layer of the parallel convolutional neural network model fits the target object to be measured in the physical process, and establishes the total variation regularization constraint loss function Loss2 as: Among them, Loss2 is the total variation regularization constraint loss function; θ x,y is the parameter value of the convolution kernel at (x, y); S33: The sparse constraint loss function Loss3 is determined by the LO norm constraint: Loss3=||θ x,y ||0 Among them, Loss3 is the sparse constraint loss function; S34: The non-negative prior is used as the final constraint condition update method: θ x,y ←max(θ x,y ,0).

8. The parallel convolution single pixel detector imaging method based on physical information mixing constraints according to claim 7, characterized in that: The element values ​​in the network output simulation value parallel matrix S' and the single pixel detection value parallel matrix S in step S31 need to be normalized with zero mean and unit variance, specifically: in, is the result of row zero mean and unit variance standardization; m k is the mean value of the kth row in the parallel matrixed detection value, that is, the mean value of the light intensity detection value in the kth frame; s k is the standard deviation of the kth row in the parallel matrixed detection values, that is, the standard deviation of the light intensity detection values ​​in the kth frame.

9. The parallel convolution single pixel detector imaging method based on physical information mixing constraints according to claim 1, characterized in that: In step S4, the physical information mixed constraint condition is used as the final loss function together with the physical information detection value loss constraint through total variation regularization and sparsity constraint. The non-negative constraint acts on the optimized convolution kernel parameters alone, so that the negative terms in the convolution kernel parameters are set to 0 to ensure the non-negative constraint while further increasing the sparsity.

Images