Real-time terahertz wave single-pixel imaging method, device and system and computer equipment

Abstract

The invention relates to the technical field of terahertz wave single-pixel imaging, and discloses a real-time terahertz wave single-pixel imaging method, device and system and computer equipment, and the method comprises the steps: obtaining a terahertz wave single-pixel measurement value of a to-be-imaged object; the terahertz wave single-pixel measurement value is a one-dimensional signal obtained by measuring an object to be imaged through a terahertz wave single-pixel detector; and inputting the terahertz wave single-pixel measurement value into a pre-constructed image reconstruction model to obtain a terahertz wave reconstruction image of the object to be imaged. Through the deep neural network model based on physical guidance, the reconstruction quality and reconstruction time of the terahertz wave image under the under-sampling condition can be effectively improved, and through the real-time terahertz wave single-pixel imaging device with a high-speed switch, the relation between the imaging speed and the imaging quality is effectively balanced, and the imaging quality is improved. And the reconstruction efficiency of real-time terahertz wave single-pixel imaging is further improved.

Description

Technical Field

[0001] The present invention relates to the technical field of terahertz wave single-pixel imaging, and in particular to a real-time terahertz wave single-pixel imaging method, device, system and computer equipment. Background Art

[0002] Terahertz waves refer to electromagnetic waves with a frequency range of 0.1-10 THz, which are characterized by being non-invasive, non-destructive and non-ionizing. Compared with X-rays, the photon energy of terahertz waves is lower, and the potential damage to biological tissues is significantly reduced. In addition, terahertz wave imaging technology can detect the internal structure information of opaque materials that are difficult to detect due to the high absorption and strong scattering between visible light and matter. Based on these characteristics, terahertz wave imaging shows broad application prospects in non-destructive detection, safety monitoring and medical diagnosis.

[0003] Currently, the commonly used terahertz wave single-pixel imaging methods include the single-pixel imaging method based on compressed sensing and the single-pixel imaging method based on deep learning. However, both methods have certain limitations. On the one hand, the method based on compressed sensing has the disadvantages of poor imaging quality and slow imaging speed. On the other hand, although the method based on deep learning can also be applied to terahertz wave single-pixel imaging, the supervised training data set of the neural network with excellent performance depends on the experimental data in the real-time terahertz wave single-pixel imaging task. The terahertz wave system is difficult to model due to the existence of various noises. The data-driven deep learning model without any physical prior is not suitable for the reconstruction of terahertz wave images and cannot give the optimal results. These data-driven deep learning strategies also have common problems such as generalization and interpretability. Therefore, the current terahertz wave single-pixel imaging method based on deep learning cannot reconstruct terahertz images with higher pixel resolution at video speed. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a real-time terahertz wave single-pixel imaging method, system, device and computer equipment, which can solve the problems of poor imaging quality and slow imaging speed in existing imaging methods and achieve the effect of terahertz wave imaging with high quality, high pixel resolution and video speed.

[0005] In a first aspect, the present invention provides a real-time terahertz wave single-pixel imaging method, the method comprising: Acquire a terahertz wave single-pixel measurement value of the object to be imaged, wherein the terahertz wave single-pixel measurement value is a one-dimensional signal obtained by measuring the object to be imaged by a terahertz wave single-pixel detector; Inputting the terahertz wave single-pixel measurement value into a pre-constructed image reconstruction model to obtain a terahertz wave reconstructed image of the object to be imaged; The image reconstruction model includes a physical coding layer and a deep neural network model, wherein the physical coding layer is used to associate the terahertz wave single pixel measurement value with the measurement matrix, generate a terahertz wave image matrix, and input the terahertz wave image matrix into the deep neural network model; The deep neural network model adopts a coding and decoding structure based on an attention gating mechanism, which is used to perform feature extraction and image reconstruction on the terahertz wave image matrix, and output the terahertz wave reconstructed image.

[0006] Furthermore, the step of training the image reconstruction model includes: Inputting a sample image into the physical coding layer, encoding the sample image through a measurement matrix to obtain a one-dimensional measurement vector, and performing a correlation analysis operation on the one-dimensional measurement vector and the measurement matrix to generate a first terahertz wave image matrix; Inputting the first terahertz wave image matrix into the deep neural network model, performing feature extraction and image reconstruction on the first terahertz wave image matrix, and outputting a first terahertz wave reconstructed image; Iteratively train the image reconstruction model according to the above steps until an iteration stop condition is reached to obtain the trained image reconstruction model, wherein the iteration stop condition includes convergence of the loss function or reaching a maximum number of iterations; The one-dimensional measurement vector is represented by the following formula: In the formula, is the nth measurement value in the one-dimensional measurement vector, * is the convolution operator, is the nth measurement matrix, is the sample image, x and y represent the image space coordinates; The first terahertz wave image matrix is ​​expressed by the following formula: In the formula, is the first terahertz wave image matrix, N is the number of measurement matrices, is the mean operation; The steps of performing image reconstruction using the trained image reconstruction model include: Inputting the terahertz wave single pixel measurement value into the physical coding layer, performing a correlation analysis operation on the terahertz wave single pixel measurement value and the measurement matrix to obtain a second terahertz wave image matrix; The second terahertz wave image matrix is ​​input into the deep neural network model, feature extraction and image reconstruction are performed on the second terahertz wave image matrix, and a terahertz wave reconstructed image corresponding to the terahertz wave single pixel measurement value is output.

[0007] Furthermore, the deep neural network model includes a downsampling encoding layer, an upsampling decoding layer and a multi-scale feature aggregation layer; The downsampling encoding layer is used to extract feature maps of different scales, concatenate the feature maps of different scales through multi-scale jump connections, and pass them to the upsampling decoding layer; The upsampling decoding layer is used to perform feature fusion on feature maps of different scales to obtain a terahertz wave aggregated feature map, and transmit it to the multi-scale feature aggregation layer; The multi-scale feature aggregation layer uses multi-scale deep supervision to learn hierarchical representation from the terahertz wave aggregate feature map; An attention gate control is provided between the upsampling decoding layer and the multi-scale feature aggregation layer, and the attention gate control is used to generate an attention weight according to the terahertz wave aggregation feature map input by the upsampling decoding layer and the gating signal input by the multi-scale feature aggregation layer, and input the attention weight into the multi-scale feature aggregation layer; Among them, the attention weight is expressed by the following formula: In the formula, F represents the attention weight, x represents the terahertz wave aggregate feature map input from the upsampling decoding layer of this layer, and g represents the gating signal input from the multi-scale feature aggregation layer of the next layer. represents matrix element addition, represents matrix element-wise multiplication, represents a one-dimensional channel attention module, Represents a two-dimensional spatial attention module.

[0008] Furthermore, the loss function of the multi-scale feature aggregation layer is the minimum value of the root mean square difference between the output of the multi-scale feature aggregation layer and the single-pixel measurement value of the terahertz wave, and the loss function of the deep neural network model is the weighted sum of the loss functions of each of the multi-scale feature aggregation layers.

[0009] Furthermore, the loss function of the multi-scale feature aggregation layer is expressed by the following formula: The loss function of the deep neural network model is expressed by the following formula: In the formula, represents the loss function of the i-th multi-scale feature aggregation layer, represents the output of the i-th multi-scale feature aggregation layer, represents the sample image, x and y represent the image space coordinates, W represents the weight parameter, represents the loss function of the deep neural network model, represents the weight parameter of the i-th loss function, represents the noise term.

[0010] In a second aspect, the present invention provides a real-time terahertz wave single-pixel imaging device, the device comprising: A terahertz wave source module, a terahertz wave spatial light modulation module and a terahertz wave detection module, wherein the object to be imaged is placed on an optical path between the terahertz wave source module and the terahertz wave spatial light modulation module; The terahertz wave spatial light modulation module includes a pump laser, a beam expansion lens, a digital micromirror device, a projection lens, an aperture and a terahertz wave modulator; The terahertz wave source module is used to emit continuous terahertz waves to the object to be imaged, and the generated terahertz wave beam carrying information of the object to be imaged is received by the terahertz wave modulator; The pump laser is used to generate a pump laser beam, and the pump laser beam is projected to the digital micromirror device through the beam expansion lens; The digital micromirror device is used to modulate and reflect the pump laser beam, and the reflected pump laser beam is received by the terahertz wave modulator through the projection lens and the aperture; The terahertz wave modulator is used to modulate the received terahertz wave beam according to the received pump laser beam to generate a modulated terahertz wave beam; The terahertz wave detection module is used to detect the modulated terahertz wave light beam and generate a terahertz wave single-pixel measurement value.

[0011] Further, the terahertz wave detection module includes a converging parabolic mirror group, a terahertz wave detector and a data acquisition module; The converging parabolic mirror group is used to converge the modulated terahertz wave light beam to the terahertz wave detector; The terahertz wave detector is a single-pixel detector, which is used to detect the modulated terahertz wave beam and generate a photoelectric signal; The data acquisition module is used to collect the photoelectric signal and convert it into a terahertz wave single-pixel measurement value.

[0012] Furthermore, the digital micromirror device is also used to modulate the pump laser beam according to a preloaded speckle image generated based on a measurement matrix, and outputs a trigger signal to the data acquisition module each time it is modulated, so that the data acquisition module performs synchronous data acquisition according to the trigger signal.

[0013] In a third aspect, the present invention provides a real-time terahertz wave single-pixel imaging system, the system comprising: A data acquisition module, used to acquire a terahertz wave single-pixel measurement value of the object to be imaged, wherein the terahertz wave single-pixel measurement value is a one-dimensional signal obtained by measuring the object to be imaged by a terahertz wave single-pixel detector; An image reconstruction module, used for inputting the terahertz wave single pixel measurement value into a pre-built image reconstruction model to obtain a terahertz wave reconstructed image of the object to be imaged; The image reconstruction model includes a physical coding layer and a deep neural network model, wherein the physical coding layer is used to associate the terahertz wave single pixel measurement value with the measurement matrix, generate a terahertz wave image matrix, and input the terahertz wave image matrix into the deep neural network model; The deep neural network model adopts a coding and decoding structure based on an attention gating mechanism, which is used to perform feature extraction and image reconstruction on the terahertz wave image matrix, and output the terahertz wave reconstructed image.

[0014] In a fourth aspect, an embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0015] The present invention provides a real-time terahertz wave single-pixel imaging method, device, system and computer equipment. The present invention constructs a deep neural network model based on physics guidance, guides the terahertz wave image reconstruction process through correlation analysis operations, and provides interpretability and physical image properties for the imaging model. Through the physical coding layer guided by physics, accurate image reconstruction priors are learned in a data-driven manner, so as to have powerful global feature and local feature modeling capabilities. An attention gate mechanism is adopted in the deep neural network model to focus on feature information related to image reconstruction, and effectively suppress the feature response of irrelevant background areas in the multi-scale feature map. At the same time, multi-scale deep supervision is adopted in the output of each multi-scale feature aggregation layer, and hierarchical representation is learned from the aggregated feature map, which enhances the generalization ability of the network, ensures the accuracy of the imaging results, and effectively improves the quality of terahertz wave image reconstruction under undersampling conditions. The present invention also provides a real-time terahertz wave single-pixel imaging device with high-speed switching, which effectively balances the relationship between imaging speed and imaging quality, and further improves the reconstruction time and reconstruction quality of real-time terahertz wave single-pixel imaging. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a schematic flow chart of a real-time terahertz wave single-pixel imaging method in an embodiment of the present invention; Figure 2 yes Figure 1 Schematic diagram of the structure of the image reconstruction model; Figure 3is a structural schematic diagram of a real-time terahertz wave single-pixel imaging device in an embodiment of the present invention; Figure 4 is another structural schematic diagram of a real-time terahertz wave single-pixel imaging device in an embodiment of the present invention; Figure 5 is a schematic diagram of experimental imaging results in an embodiment of the present invention; Figure 6 is a structural schematic diagram of a real-time terahertz wave single-pixel imaging system in an embodiment of the present invention; Figure 7 It is a diagram of the internal structure of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] See also Figure 1 A real-time terahertz wave single-pixel imaging method proposed in the first embodiment of the present invention includes steps S10 to S20: Step S10, obtaining a terahertz wave single-pixel measurement value of the object to be imaged, wherein the terahertz wave single-pixel measurement value is a one-dimensional signal obtained by measuring the object to be imaged using a terahertz wave single-pixel detector; Step S20: inputting the terahertz wave single-pixel measurement value into a pre-constructed image reconstruction model to obtain a terahertz wave reconstructed image of the object to be imaged.

[0019] In view of the problems of poor imaging quality and slow imaging speed in the current terahertz wave single-pixel imaging method based on deep learning, the present invention provides a physics-guided deep neural network model for terahertz wave image reconstruction. The structure of the image reconstruction model is described in detail below.

[0020] See also Figure 2In this embodiment, the image reconstruction model includes an upstream physical coding for physical guidance and a downstream deep neural network model, wherein the role of the physical coding layer is to provide physical image guidance for the downstream deep neural network model, thereby enhancing the generalization ability and interpretability of the model, and the role of the deep neural network model is to further extract features and reconstruct the image matrix output by the physical coding layer, thereby obtaining a high-precision reconstructed image. When training the image reconstruction model, the physical coding layer and the deep neural network model of the image reconstruction model are trained uniformly, and the specific training steps include: Inputting a sample image into the physical coding layer, encoding the sample image through a measurement matrix to obtain a one-dimensional measurement vector, and performing a correlation analysis operation on the one-dimensional measurement vector and the measurement matrix to generate a first terahertz wave image matrix; Inputting the first terahertz wave image matrix into the deep neural network model, performing feature extraction and image reconstruction on the first terahertz wave image matrix, and outputting a first terahertz wave reconstructed image; The image reconstruction model is iteratively trained according to the above steps until an iteration stop condition is reached to obtain the trained image reconstruction model. The iteration stop condition includes the convergence of the loss function or the reaching of the maximum number of iterations.

[0021] In this embodiment, the processing process of the physical coding layer includes the coding processing of the input sample image and the measurement matrix, and the correlation analysis processing of the one-dimensional measurement vector obtained by coding and the measurement matrix, that is, it is divided into two stages: coding processing and correlation analysis. Among them, the coding processing stage is applied in the training stage of the image reconstruction model. During training, the data set selected in this embodiment is a data set based on sample images, such as the STL10 image data set. When the sample image in the data set is input into the physical coding layer, the sample image is encoded by the measurement matrix to obtain a one-dimensional measurement vector, and then the one-dimensional measurement vector obtained by encoding is subjected to correlation analysis processing with the measurement matrix. In other words, the one-dimensional measurement vector obtained in the coding processing stage in the physical coding layer is actually a simulation of the terahertz wave single-pixel measurement value obtained by measuring the imaging object through the terahertz wave single-pixel detector. Through coding processing, it is not necessary to use a large number of terahertz wave single-pixel measurement values ​​to construct the training data set, but directly use the existing data set based on sample images to train the model, thereby improving the training efficiency of the model. During a training process, the terahertz wave image matrix output by the physical coding layer will be input into the deep neural network model, and feature extraction and image reconstruction will be performed in the deep neural network model to complete an iterative training. When the loss function of the image reconstruction model reaches the convergence condition or reaches the maximum number of iterations, the iterative training stops and a trained image reconstruction model is obtained.

[0022] After the image reconstruction model is trained, the terahertz wave single pixel measurement value is input into the physical coding layer of the trained image reconstruction model, and the terahertz wave single pixel measurement value is correlated with the strategy matrix to generate a terahertz wave image matrix; that is, in the actual terahertz wave single pixel reconstruction process, the processing of the terahertz wave single pixel measurement value in the physical coding layer starts with correlation analysis, and the terahertz wave single pixel measurement value is used as Figure 2 The one-dimensional measurement vector in , that is, the measurement value, is correlated with the measurement matrix to obtain the terahertz wave image matrix, and then the terahertz wave image matrix is ​​input into the deep neural network model for feature extraction and image reconstruction to obtain the terahertz wave reconstructed image.

[0023] For the image reconstruction model in a training process, the processing steps of the model in the physical coding layer are explained: When training the image reconstruction model, the sample images in the data set are input into the physical coding layer, denoted as , and then encode the sample image through the measurement matrix to obtain a one-dimensional measurement vector: In the formula, is the nth measurement value in the one-dimensional measurement vector, * is the convolution operator, is the nth measurement matrix, is the sample image, x and y represent the image space coordinates.

[0024] In this embodiment, the measurement matrix is ​​in the form of two mutually opposite measurement matrices to reduce the influence of noise in the measurement, and its formula is expressed as follows: In the formula, represents the positive mask matrix of the nth measurement matrix, Represents the inverse mask matrix of the nth measurement matrix.

[0025] In a preferred embodiment, the present invention uses a Hadamard basis matrix composed of 1 and -1 as a measurement matrix, and its mathematical matrix in the second-order difference form is: Of course, other matrices such as random Gaussian matrix, Fourier basis matrix, deep learning basis matrix, etc. can also be used as measurement matrices, and no excessive restrictions are made here.

[0026] Then the one-dimensional measurement vector is Figure 2The measured values ​​in are correlated and analyzed with the measurement matrix to obtain the suboptimal terahertz wave image matrix. In order to distinguish it from the terahertz wave image matrix output by the physical coding layer in the actual reconstruction process, the terahertz wave image matrix output in the training process is used as the first terahertz wave image matrix. The mathematical transfer function of this process can be expressed as: Where N is the number of measurement matrices, is the mean operation, is the first terahertz wave image matrix, and is used as the physical coding layer of physical guidance in this embodiment. The association analysis process of this embodiment can be any one of ghost imaging, differential ghost imaging, normalized ghost imaging, Hadamard single pixel imaging, Fourier single pixel imaging or compressed sensing single pixel imaging based on iterative operation.

[0027] In this embodiment, the terahertz wave image reconstruction process is guided by the physical coding layer, providing interpretability and physical image properties for the subsequent deep neural network model, thereby improving the reconstruction quality and efficiency of the terahertz wave image.

[0028] The terahertz wave image matrix output by the physical coding layer will be input into the deep neural network model for further feature extraction and image reconstruction. In this embodiment, the deep neural network model adopts a coding and decoding structure based on the attention gating mechanism, including a downsampling coding layer, an upsampling decoding layer and a multi-scale feature aggregation layer. The output of the physical coding layer is input into the downsampling coding layer. In this embodiment, multiple downsampling coding layers with different parameters, that is, multiple convolutional pooling layers with different parameters are provided. Feature maps of different scales are extracted through multiple downsampling coding layers. These feature maps of different sizes are spliced ​​through multi-scale jump connections and transmitted to the upsampling decoding layer. Each upsampling coding layer fuses feature maps of smaller scales and the same scale, thereby obtaining a multi-scale terahertz wave aggregation feature map.

[0029] by Figure 2 As an example, the multi-scale skip connection is briefly described. Assume that the downsampling coding layers are X from top to bottom. En1 To X En5 , the upsampling decoding layers are X from top to bottom. De1 To X De5 , where X En5 and X De5 Represented by the same layer, with decoder X De3 For example, the source of its feature map is divided into three parts. The first part is the small-scale feature map of the encoder, that is, X De1 and X De2The feature map can be reduced in size through non-overlapping maximum pooling operations, and the number of channels of the feature map can be changed through convolution operations. The second part is the feature map of the same scale in the encoder, that is, X De3 The third part is the large-scale feature map in the decoder, which is the lowest dimension X in the figure. De5 The feature map of the next scale is X, and the size of the feature map is enlarged by upsampling operation, that is, bilinear interpolation. The size and number of channels of the feature map are changed by convolution operation. Finally, four feature maps with the same number of channels are obtained. De4 It is used as the gating signal for attention gate control. The size of the feature map is enlarged by upsampling operation, i.e. bilinear interpolation. The size and number of channels of the feature map are changed by convolution operation. Finally, X is obtained by feature aggregation. En3 The feature map of the image is obtained by combining the coarse-level and fine-level dense connections in this embodiment, thereby realizing multi-scale feature fusion.

[0030] In this embodiment, a corresponding multi-scale feature aggregation layer is also provided. Each multi-scale feature aggregation layer adopts deep supervision, learns hierarchical representation from the multi-scale terahertz wave aggregation feature map, and obtains a reconstructed image by aggregating the feature map input by the upsampling decoding layer. In this embodiment, the deep supervision is preferably set to a 3×3 convolution layer, which is equivalent to de-supervising the output of each multi-scale feature aggregation layer. After the introduction of deep supervision, each sub-network of the model, i.e., L1~L4, can output the image separately. After training, only the output of the multi-scale feature aggregation layer L1 is needed as the final reconstructed terahertz wave image, thereby reducing the model structure and improving the output efficiency of the model. By introducing deep supervision, the generalization ability of the deep neural network in this embodiment is enhanced, and the terahertz wave image is guided to be reconstructed more accurately. It should be noted that the specific steps of the multi-scale jump connection in this embodiment can refer to the conventional multi-scale jump connection, and will not be repeated here.

[0031] In a preferred embodiment, this embodiment also uses attention gate control to focus on feature information related to terahertz wave image reconstruction, thereby suppressing feature responses in irrelevant background areas in the multi-scale feature map, wherein the attention gate control is set between the upsampling decoding layer and the multi-scale feature aggregation layer, and the attention mechanism formula used can be expressed as: In the formula, F represents the attention weight, x represents the terahertz wave aggregate feature map input from the upsampling decoding layer of this layer, and g represents the gating signal input from the multi-scale feature aggregation layer of the next layer. represents matrix element addition, represents matrix element-wise multiplication, represents a one-dimensional channel attention module, Represents a two-dimensional spatial attention module.

[0032] It should be noted here that since there is one less multi-scale feature aggregation layer than the upsampling decoding layer, the gating signal input into each attention gate control comes from the multi-scale feature aggregation layer of the next layer.

[0033] When training the image reconstruction model, this embodiment preferably uses the Adam optimizer for optimization, and takes the minimum value of the root mean square difference between the output of each multi-scale feature aggregation layer and the sample image as the loss function, which is expressed as follows: In the formula, represents the loss function of the i-th multi-scale feature aggregation layer, represents the output of the i-th multi-scale feature aggregation layer, represents the sample image, x and y represent the image space coordinates, and W represents the weight parameter.

[0034] At the same time, the training process of the model is represented as multi-task regression, and a multi-task learning strategy is used to balance these weights. The weighted sum of the loss functions of the four multi-scale feature aggregation layers is used to obtain the final loss function, which is expressed by the following formula: In the formula, represents the loss function of the deep neural network model, represents the weight parameter of the i-th loss function, represents the noise term.

[0035] In the image reconstruction model, each multi-scale feature aggregation layer outputs an undersampled terahertz wave image. With sample input As the mapping sample data pair, the proposed network parameters are optimized and updated through the continuous convergence of the above loss function network. The loss function tends to converge and minimize, and the trained model is obtained. The multi-scale feature aggregation layer The output is used as the final reconstructed terahertz wave image. In order to prevent overfitting due to too many iterations and ensure the stability of training, preferably, the preset maximum number of iterations can be used as the iteration stop condition for model training.

[0036] When the trained image reconstruction model is used to reconstruct the image of the terahertz wave single pixel measurement value, the processing steps of the model include: Inputting the terahertz wave single pixel measurement value into the physical coding layer, performing a correlation analysis operation on the terahertz wave single pixel measurement value and the measurement matrix to obtain a second terahertz wave image matrix; The second terahertz wave image matrix is ​​input into the deep neural network model, feature extraction and image reconstruction are performed on the second terahertz wave image matrix, and a terahertz wave reconstructed image corresponding to the terahertz wave single pixel measurement value is output.

[0037] When using the trained image reconstruction model to reconstruct the image, the input terahertz wave single-pixel measurement value can be regarded as a one-dimensional measurement vector obtained by encoding the sample image in the above model training process. At this time, the terahertz wave single-pixel measurement value and the measurement matrix are associated and analyzed, and the generated second terahertz wave image matrix is ​​input into the deep neural network model for feature extraction and image reconstruction, so as to obtain the terahertz wave reconstructed image corresponding to the terahertz wave single-pixel measurement value. The specific processing steps in the actual reconstruction process can refer to the processing steps in the above training process, which will not be repeated here.

[0038] This embodiment constructs a deep neural network model based on physics guidance, guides the terahertz wave image reconstruction process through correlation analysis operations, and provides interpretability and physical image properties for the imaging model. Through the physical coding layer guided by physics, accurate image reconstruction priors are learned in a data-driven manner, so as to have powerful global and local feature modeling capabilities. The attention gate mechanism is adopted in the deep neural network model to focus on the feature information related to image reconstruction, and effectively suppress the feature response of irrelevant background areas in the multi-scale feature map. At the same time, multi-scale deep supervision is adopted in the output of each multi-scale feature aggregation layer, and the hierarchical representation is learned from the aggregated feature map, which enhances the generalization ability of the network, ensures the accuracy of the imaging results, and effectively improves the quality of terahertz wave image reconstruction under undersampling conditions. After the model training is completed, high-quality images can be reconstructed directly from undersampled one-dimensional terahertz wave measurements, avoiding time-consuming iterative optimization steps. Thereby effectively improving the efficiency and quality of terahertz wave single-pixel image reconstruction.

[0039] In practical applications, according to the needs of terahertz wave imaging tasks, this embodiment uses 10,000 images in the STL10 image data set to train and verify the model. All images are converted into grayscale images, and the resolution is adjusted to 64×64 to adapt to the reconstructed terahertz wave images with a pixel resolution of 64×64. The image reconstruction inference time of the trained model on the RTX 3060 graphics processor is 20 milliseconds / frame, which meets the requirement that the video imaging speed is at least about 41 milliseconds / frame. High-quality terahertz wave images with a pixel resolution of 64×64 can also be reconstructed under the condition of 128 measurements. The image reconstruction model provided by the present invention not only reduces the dependence on high-performance computing resources, but also reduces the energy consumption of the system, and can also operate efficiently in an environment with limited computing and storage resources.

[0040] In order to further improve the speed of real-time terahertz wave single-pixel imaging, based on the real-time terahertz wave single-pixel imaging method based on the image reconstruction model provided by the present invention, the second embodiment of the present invention proposes a real-time terahertz wave single-pixel imaging device, please refer to Figure 3 The device includes: a terahertz wave source module 20, a terahertz wave spatial light modulation module 30 and a terahertz wave detection module 40, and the object 10 to be imaged is placed on the optical path between the terahertz wave source module 20 and the terahertz wave spatial light modulation module 30. The terahertz wave spatial light modulation module 30 includes a pump laser 31, a beam expansion lens 32, a digital micromirror device 33, a projection lens 34, an aperture 35 and a terahertz wave modulator 36.

[0041] The device is used to modulate the object to be imaged and generate a single-pixel measurement value of a terahertz wave. Specifically, after the terahertz wave interacts with the object to be imaged, the terahertz wave carrying information of the object to be imaged is encoded, and the amplitude of the modulated light field is collected through a single-pixel terahertz wave detector.

[0042] In a preferred embodiment, the terahertz wave detection module 40 includes a converging parabolic mirror group 41, a terahertz wave detector 42 and a data acquisition module 43. The imaging optical path of the present device is described below based on the above structure. In this embodiment, the terahertz wave source module 20 is used to output continuous terahertz waves. The terahertz wave source module 20 is adjacent to the object module 10 to be imaged. The continuous terahertz wave interacts with the object to be imaged in a transmission or reflection manner, and the generated terahertz wave beam carries the information of the object to be imaged. The terahertz wave spatial light modulation module 30 is adjacent to the object module 10 to be imaged, and the pump laser 31 therein is used to generate a pump laser beam. The pump laser beam is collimated and expanded after passing through the beam expansion lens 32 to completely cover the mirror surface of the digital micromirror device 33. The digital micromirror device 33 is used to receive the expanded pump laser beam. The pump laser beam is modulated according to the speckle pattern preset by the measurement matrix and then passes through the projection lens. 34 and the aperture 35 are transmitted to the terahertz wave modulator 36; at the same time, the terahertz wave modulator 36 is also used to receive the terahertz wave light beam carrying the information of the object to be imaged, and modulate the terahertz wave light beam based on the pump laser beam, and the modulated terahertz wave light beam is transmitted to the terahertz wave detection module 40. The converging parabolic mirror group 41 in the terahertz wave detection module 40 is used to converge the modulated terahertz wave light beam to the terahertz wave detector 42. The terahertz wave detector 42 is a single-pixel detector, which generates a photoelectric signal under the excitation of the terahertz wave; the data acquisition module 43 is connected to the terahertz wave detector 42, and converts the electrical signal of the detector into a digital signal that can be used by the image reconstruction model, so as to realize real-time terahertz wave single-pixel imaging of the object to be imaged 10 according to the data signal.

[0043] See also Figure 4In a preferred embodiment, the terahertz wave source module 20 outputs continuous terahertz waves through a micro-signal generator and a terahertz wave frequency doubling module; the pump laser 31 uses a 532 nm continuous wave pump laser beam to generate electron-hole pairs of a terahertz wave modulator 36, thereby achieving the purpose of modulating the terahertz wave beam. The terahertz wave modulator 36 may include passivated silicon, passivated germanium, a single crystal germanium wafer, a high-resistance silicon wafer, sapphire or vanadium dioxide. The specific implementation may be selected according to actual conditions, and no excessive restrictions are made here. The digital micromirror device 33 uses a DMD spatial light modulator composed of 768×1024 13.68 micron×13.68 micron small micromirrors. The digital micromirror device 33 is preloaded with a series of speckle images generated by the measurement matrix selected according to the image reconstruction model by the computer, and the pump laser beam is modulated by the speckle image. The modulated pump laser beam is projected to the terahertz wave modulator through a projection lens and an aperture (not shown), wherein the converging parabolic mirror 41 group includes a first off-axis parabolic mirror and a second off-axis parabolic mirror. The object mirror and the first off-axis parabolic mirror are arranged between the digital micromirror device 33 and the terahertz wave modulator 36. The modulated pump laser beam is projected to the terahertz wave modulator 36 through the small hole in the middle of the first off-axis parabolic mirror, and the modulated terahertz wave beam generated by the terahertz wave modulator 36 is reflected to the terahertz wave single pixel detector through the reflection of the first off-axis parabolic mirror and the second off-axis parabolic mirror. During each projection process of the DMD, a rising edge trigger signal is output for synchronously triggering the data acquisition module 43 to collect. The data acquisition module 43 uses an ADC data acquisition card with an acquisition bit width of 16 bits, a maximum sampling rate of 2 megaseconds, and an accuracy of 1.66 millivolts.

[0044] Since the flip frequency of the digital micromirror device 33 can be up to 20 kHz for stable operation, and the modulation speed of the terahertz wave modulator 36 is limited by its own material properties and the DMD switching speed, in order to make the terahertz wave modulator 36 work at 20 kHz, it is required that data acquisition and handling must be completed within 50 microseconds. The response time of the terahertz wave detector 42 must be less than 50 microseconds, so this embodiment preferably uses an AlGaN / GaN high electron mobility transistor for the terahertz wave detector 42, and the response time is 0.2 microseconds. The detector has an integrated photoelectric conversion function, which can convert the terahertz wave intensity information into an analog voltage output. After the number of projections of the speckle pattern in the digital micromirror device 33 reaches a set projection cycle threshold, the image reconstruction model can be started by the computer to realize a frame of terahertz wave image of the object to be imaged.

[0045] The real-time terahertz wave single-pixel imaging device proposed in this embodiment has the characteristics of simple structure and low cost. The terahertz continuous source used in this device has the advantages of high power density, high integration, small size, etc. The device integrates AlGaN / GaN high electron mobility transistors with a response time of 0.2 microseconds, and uses a terahertz wave modulator working under the pumping of a 532nm continuous wave pump laser to achieve a switching speed of 20 kHz. The real-time terahertz wave single-pixel imaging method proposed in the present invention is combined with a device for terahertz wave modulation of the imaged object, which can achieve video-speed-level terahertz wave single-pixel image reconstruction, while ensuring the high quality of the reconstructed image.

[0046] Figure 5 The results of terahertz single-pixel imaging with a resolution of 64×64 pixels based on the method and device provided by the present invention at different measurement times and different terahertz modulator switching speeds are shown. The experimental results show that the device provided by the present invention uses 128 terahertz wave single-pixel measurements to reconstruct a terahertz wave image with a pixel resolution of 64×64 at a switching speed of 20 kHz. At this time, the acquisition time is 6.4 milliseconds, that is, the recording speed of the terahertz wave image is 156 frames / second. The real-time terahertz wave single-pixel imaging method proposed by the present invention requires a reconstruction time of 20 milliseconds. In order to obtain the best balance between reconstruction quality and acquisition time, the present invention finally selects a switching speed of 15.625 kHz modulation speed and a single-pixel measurement with a measurement number of 512. At this time, the acquisition time of the device is about 33 milliseconds, that is, the recording speed of the terahertz wave image is 30 frames / second, which matches the reconstruction time of the real-time terahertz wave single-pixel imaging method.

[0047] See also Figure 6 Based on the same inventive concept, a real-time terahertz wave single-pixel imaging system proposed in the third embodiment of the present invention includes: A data acquisition module 1 is used to acquire a terahertz wave single-pixel measurement value of the object to be imaged, wherein the terahertz wave single-pixel measurement value is a one-dimensional signal obtained by measuring the object to be imaged through a terahertz wave single-pixel detector; An image reconstruction module 2 is used to input the terahertz wave single pixel measurement value into a pre-built image reconstruction model to obtain a terahertz wave reconstructed image of the object to be imaged; The image reconstruction model includes a physical coding layer and a deep neural network model, wherein the physical coding layer is used to associate the terahertz wave single pixel measurement value with the measurement matrix, generate a terahertz wave image matrix, and input the terahertz wave image matrix into the deep neural network model; The deep neural network model adopts a coding and decoding structure based on an attention gating mechanism, which is used to perform feature extraction and image reconstruction on the terahertz wave image matrix, and output the terahertz wave reconstructed image.

[0048] The technical features and technical effects of the real-time terahertz wave single-pixel imaging system proposed in the embodiment of the present invention are the same as those of the method proposed in the embodiment of the present invention, and are not described in detail here. Each module in the above-mentioned real-time terahertz wave single-pixel imaging system can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0049] In addition, an embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0050] See also Figure 7 , an internal structure diagram of a computer device in an embodiment, the computer device can specifically be a terminal or a server. The computer device includes a processor, a memory, a network interface, a display and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a real-time terahertz wave single-pixel imaging method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0051] It can be understood by those skilled in the art that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computing device may include more or fewer components than shown in the figure, or combine certain components, or have the same component arrangement.

[0052] In summary, the embodiments of the present invention propose a real-time terahertz wave single-pixel imaging method, device, system and computer equipment, the method obtains the terahertz wave single-pixel measurement value of the object to be imaged, the terahertz wave single-pixel measurement value is a one-dimensional signal obtained by measuring the object to be imaged through a terahertz wave single-pixel detector; the terahertz wave single-pixel measurement value is input into a pre-constructed image reconstruction model to obtain a terahertz wave reconstructed image of the object to be imaged; wherein the image reconstruction model includes a physical coding layer and a deep neural network model, the physical coding layer is used to associate the terahertz wave single-pixel measurement value with the measurement matrix, generate a terahertz wave image matrix, and input it into the deep neural network model; the deep neural network model adopts a coding and decoding structure based on an attention gating mechanism, which is used to extract features and reconstruct images of the terahertz wave image matrix, and output the terahertz wave reconstructed image. The present invention constructs a deep neural network model based on physics guidance, guides the terahertz wave image reconstruction process through association analysis operations, and provides interpretability and physical image properties for the imaging model. Through the physically guided physical coding layer, accurate image reconstruction priors are learned in a data-driven manner, thus possessing powerful global and local feature modeling capabilities. An attention gate mechanism is used in the deep neural network model to focus on feature information related to image reconstruction, effectively suppressing feature responses of irrelevant background areas in the multi-scale feature map. At the same time, multi-scale deep supervision is used in the output of each multi-scale feature aggregation layer to learn hierarchical representations from the aggregated feature map, thereby enhancing the generalization ability of the network, ensuring the accuracy of the imaging results, and effectively improving the quality of terahertz wave image reconstruction under undersampling conditions. The present invention also provides a real-time terahertz wave single-pixel imaging device with high-speed switching, which effectively balances the relationship between imaging speed and imaging quality, and further improves the reconstruction time and reconstruction quality of real-time terahertz wave single-pixel imaging.

[0053] Each embodiment in this specification is described in a progressive manner, and the same or similar parts of each embodiment can be directly referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. It should be noted that the technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above-mentioned embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0054] The above-mentioned embodiments only express several preferred implementation modes of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in the technical field, several improvements and substitutions can be made without departing from the technical principles of the present invention, and these improvements and substitutions should also be regarded as the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be based on the protection scope of the claims.

Claims

1. A real-time terahertz wave single-pixel imaging method, characterized in that: include: Acquire a terahertz wave single-pixel measurement value of the object to be imaged, wherein the terahertz wave single-pixel measurement value is a one-dimensional signal obtained by measuring the object to be imaged by a terahertz wave single-pixel detector; Inputting the terahertz wave single-pixel measurement value into a pre-constructed image reconstruction model to obtain a terahertz wave reconstructed image of the object to be imaged; The image reconstruction model includes a physical coding layer and a deep neural network model, wherein the physical coding layer is used to associate the terahertz wave single pixel measurement value with the measurement matrix, generate a terahertz wave image matrix, and input the terahertz wave image matrix into the deep neural network model; The deep neural network model adopts a coding and decoding structure based on an attention gating mechanism, which is used to perform feature extraction and image reconstruction on the terahertz wave image matrix, and output the terahertz wave reconstructed image.

2. The real-time terahertz wave single-pixel imaging method according to claim 1, characterized in that: The step of training the image reconstruction model comprises: Inputting a sample image into the physical coding layer, encoding the sample image through a measurement matrix to obtain a one-dimensional measurement vector, and performing a correlation analysis operation on the one-dimensional measurement vector and the measurement matrix to generate a first terahertz wave image matrix; Inputting the first terahertz wave image matrix into the deep neural network model, performing feature extraction and image reconstruction on the first terahertz wave image matrix, and outputting a first terahertz wave reconstructed image; Iteratively train the image reconstruction model according to the above steps until an iteration stop condition is reached to obtain the trained image reconstruction model, wherein the iteration stop condition includes convergence of the loss function or reaching a maximum number of iterations; The one-dimensional measurement vector is represented by the following formula: In the formula, is the nth measurement value in the one-dimensional measurement vector, * is the convolution operator, is the nth measurement matrix, is the sample image, x and y represent the image space coordinates; The first terahertz wave image matrix is ​​expressed by the following formula: In the formula, is the first terahertz wave image matrix, N is the number of measurement matrices, is the mean operation; The steps of performing image reconstruction using the trained image reconstruction model include: Inputting the terahertz wave single pixel measurement value into the physical coding layer, performing a correlation analysis operation on the terahertz wave single pixel measurement value and the measurement matrix to obtain a second terahertz wave image matrix; The second terahertz wave image matrix is ​​input into the deep neural network model, feature extraction and image reconstruction are performed on the second terahertz wave image matrix, and a terahertz wave reconstructed image corresponding to the terahertz wave single pixel measurement value is output.

3. The real-time terahertz wave single-pixel imaging method according to claim 1, characterized in that: The deep neural network model includes a downsampling encoding layer, an upsampling decoding layer and a multi-scale feature aggregation layer; The downsampling encoding layer is used to extract feature maps of different scales, concatenate the feature maps of different scales through multi-scale jump connections, and pass them to the upsampling decoding layer; The upsampling decoding layer is used to perform feature fusion on feature maps of different scales to obtain a terahertz wave aggregated feature map, and transmit it to the multi-scale feature aggregation layer; The multi-scale feature aggregation layer uses multi-scale deep supervision to learn hierarchical representation from the terahertz wave aggregate feature map; An attention gate control is provided between the upsampling decoding layer and the multi-scale feature aggregation layer, and the attention gate control is used to generate an attention weight according to the terahertz wave aggregation feature map input by the upsampling decoding layer and the gating signal input by the multi-scale feature aggregation layer, and input the attention weight into the multi-scale feature aggregation layer; Among them, the attention weight is expressed by the following formula: In the formula, F represents the attention weight, x represents the terahertz wave aggregate feature map input from the upsampling decoding layer of this layer, and g represents the gating signal input from the multi-scale feature aggregation layer of the next layer. represents matrix element addition, represents matrix element-wise multiplication, represents a one-dimensional channel attention module, Represents a two-dimensional spatial attention module.

4. The real-time terahertz wave single-pixel imaging method according to claim 3, characterized in that: The loss function of the multi-scale feature aggregation layer is the minimum value of the root mean square difference between the output of the multi-scale feature aggregation layer and the single-pixel measurement value of the terahertz wave, and the loss function of the deep neural network model is the weighted sum of the loss functions of each of the multi-scale feature aggregation layers.

5. The real-time terahertz wave single-pixel imaging method according to claim 4, characterized in that: The loss function of the multi-scale feature aggregation layer is expressed by the following formula: The loss function of the deep neural network model is expressed by the following formula: In the formula, represents the loss function of the i-th multi-scale feature aggregation layer, represents the output of the i-th multi-scale feature aggregation layer, represents the sample image, x and y represent the image space coordinates, W represents the weight parameter, represents the loss function of the deep neural network model, represents the weight parameter of the i-th loss function, represents the noise term.

6. A real-time terahertz wave single-pixel imaging device, characterized in that: The device is used to modulate the object to be imaged in the method according to any one of claims 1 to 5 to generate a single-pixel measurement value of a terahertz wave, and the device comprises: A terahertz wave source module, a terahertz wave spatial light modulation module and a terahertz wave detection module, wherein the object to be imaged is placed on an optical path between the terahertz wave source module and the terahertz wave spatial light modulation module; The terahertz wave spatial light modulation module includes a pump laser, a beam expansion lens, a digital micromirror device, a projection lens, an aperture and a terahertz wave modulator; The terahertz wave source module is used to emit continuous terahertz waves to the object to be imaged, and the generated terahertz wave beam carrying information of the object to be imaged is received by the terahertz wave modulator; The pump laser is used to generate a pump laser beam, and the pump laser beam is projected to the digital micromirror device through the beam expansion lens; The digital micromirror device is used to modulate and reflect the pump laser beam, and the reflected pump laser beam is received by the terahertz wave modulator through the projection lens and the aperture; The terahertz wave modulator is used to modulate the received terahertz wave beam according to the received pump laser beam to generate a modulated terahertz wave beam; The terahertz wave detection module is used to detect the modulated terahertz wave light beam and generate a terahertz wave single-pixel measurement value.

7. The real-time terahertz wave single-pixel imaging device according to claim 6, characterized in that: The terahertz wave detection module includes a converging parabolic mirror group, a terahertz wave detector and a data acquisition module; The converging parabolic mirror group is used to converge the modulated terahertz wave light beam to the terahertz wave detector; The terahertz wave detector is a single-pixel detector, which is used to detect the modulated terahertz wave beam and generate a photoelectric signal; The data acquisition module is used to collect the photoelectric signal and convert it into a terahertz wave single-pixel measurement value.

8. The real-time terahertz wave single-pixel imaging device according to claim 7, characterized in that: The digital micromirror device is also used to modulate the pump laser beam according to a preloaded speckle image generated based on a measurement matrix, and outputs a trigger signal to the data acquisition module each time it is modulated, so that the data acquisition module performs synchronous data acquisition according to the trigger signal.

9. A real-time terahertz wave single-pixel imaging system, characterized in that: The system is applied to the method according to any one of claims 1 to 5, and the system comprises: A data acquisition module, used to acquire a terahertz wave single-pixel measurement value of the object to be imaged, wherein the terahertz wave single-pixel measurement value is a one-dimensional signal obtained by measuring the object to be imaged by a terahertz wave single-pixel detector; An image reconstruction module, used for inputting the terahertz wave single pixel measurement value into a pre-built image reconstruction model to obtain a terahertz wave reconstructed image of the object to be imaged; The image reconstruction model includes a physical coding layer and a deep neural network model, wherein the physical coding layer is used to associate the terahertz wave single pixel measurement value with the measurement matrix, generate a terahertz wave image matrix, and input the terahertz wave image matrix into the deep neural network model; The deep neural network model adopts a coding and decoding structure based on an attention gating mechanism, which is used to perform feature extraction and image reconstruction on the terahertz wave image matrix, and output the terahertz wave reconstructed image.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

Images