Real-time imaging method and apparatus, electronic device, and storage medium
Through deep learning, the image reconstruction network is built, and the mask parameters are trained using the training set to optimize the image signal-to-noise ratio, which solves the problem of poor imaging quality caused by the small number of masks in terahertz imaging, and achieves high-quality imaging under a small number of masks .
Patent Information
- Application Number
- PCT/CN2023/136598
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-11
- Filing Date
- 2023-12-05
- Publication Date
- 2024-11-14
AI Technical Summary
In the field of terahertz imaging, traditional algorithms cannot ensure imaging quality while reducing the number of masks, resulting in poor imaging quality under the condition of small masks.
Build an image reconstruction network through deep learning, use training images in the training set for training, obtain mask parameters, and acquire preprocessed mask images to realize the design of projection mask under undersampling conditions and optimize image signal-to-noise Improve imaging quality.
It realizes the reconstruction of high-quality images under a small amount of masks, effectively improving the imaging quality, and solving the problem of poor imaging quality caused by the small number of masks.
Smart Images

Figure CN2023136598_14112024_PF_FP_ABST
Abstract
Description
Real-time imaging method, device, electronic device and storage medium Technical Field
[0001] The present application relates to the field of real-time imaging technology. Specifically, the present application relates to a real-time imaging method, device, electronic device and storage medium. Background Art
[0002] Terahertz waves, due to their unique wavelengths, possess unique properties. In recent years, they have been widely applied in materials, biology, medicine, imaging, and other fields. Among these, single-pixel computational imaging (THz imaging) holds the greatest research potential. Specifically, a single-pixel detector measures the reflected or transmitted light from a target illuminated by a coded THz wave to reconstruct an image of the target.
[0003] However, in the field of terahertz imaging, the quality of terahertz single-pixel imaging is positively correlated with the number of coding masks, and traditional algorithms cannot guarantee imaging quality while reducing the number of masks.
[0004] From the above, we can see that how to improve imaging quality with a small number of masks remains to be solved.
[0005] Summary of the Invention
[0006] This application provides a real-time imaging method, device, electronic device, and storage medium, which can solve the problem of poor imaging quality when the number of masks is small in the related art. The technical solution is as follows:
[0007] According to one aspect of the present application, a real-time imaging method includes: obtaining a training set; the training set includes multiple training images; using the training images in the training set to train an image reconstruction network containing mask parameters, and obtaining multiple mask images based on the mask parameters in the trained image reconstruction network; preprocessing the multiple mask images, using the preprocessed mask images to perform image acquisition in a terahertz single-pixel acquisition system to obtain an input image of the target; inputting the input image of the target into the image reconstruction network for image reconstruction to obtain a reconstructed image of the target.
[0008] According to one aspect of the present application, a real-time imaging device includes: a training set acquisition module for acquiring a training set; the training set includes multiple training images; a training module for training an image reconstruction network containing mask parameters using the training images in the training set, and obtaining multiple mask images based on the mask parameters in the trained image reconstruction network; an image acquisition module for preprocessing the multiple mask images, and using the preprocessed mask images to perform image acquisition in a terahertz single-pixel acquisition system to obtain an input image of the target; and an image reconstruction module for inputting the input image of the target into the image reconstruction network for image reconstruction to obtain a reconstructed image of the target.
[0009] In an exemplary embodiment, the training module includes: a training unit, used to input a current training image in the training set into the image reconstruction network for training to obtain an output image corresponding to the current training image; a calculation unit, used to calculate a loss value based on the difference between the current training image and its corresponding output image; a judgment unit, used to complete the training and obtain the image reconstruction network if the loss value meets the convergence condition; otherwise, update the parameters of the image reconstruction network and obtain the next training image in the training set to continue training the image reconstruction network until the loss value meets the convergence condition; the parameters of the image reconstruction network include the mask parameters.
[0010] In an exemplary embodiment, the image reconstruction module includes: an encoding unit, used to input the input image of the target into the encoding layer, and use the mask parameters in the encoding layer to simulate single-pixel sampling of the input image to obtain an encoding result; a singular value decomposition unit, used to input the encoding result into the decoding layer for singular value decomposition to obtain a decoding result; and an enhancement unit, used to input the decoding result into the enhancement layer for enhancement processing to obtain a reconstructed image of the target.
[0011] In an exemplary embodiment, the enhancement layer includes multiple downsampling layers, multiple upsampling layers, and multiple residual connection layers respectively connected between the downsampling layers and the upsampling layers; the enhancement unit includes: a feature extraction subunit, used to extract features from the decoding results through multiple downsampling layers to obtain multiple input features; a transmission subunit, used to perform feature propagation between the connected upsampling layers and downsampling layers through multiple residual connection layers, and transmit the multiple input features from the downsampling layers to the corresponding upsampling layers respectively; a feature fusion subunit, used to perform feature fusion on the multiple input features through multiple upsampling layers to obtain a reconstructed image of the target.
[0012] In an exemplary embodiment, the image acquisition module includes: an interpolation unit, configured to interpolate a plurality of mask images according to a set image size, and modulate the interpolated mask images to obtain a preprocessed mask image.
[0013] In an exemplary embodiment, the device further includes: a projection module for projecting the preprocessed mask image onto the target using terahertz waves in the terahertz single-pixel acquisition system, so that the target generates a modulation signal; and an acquisition module for obtaining an input image of the target by acquiring the modulation signal.
[0014] According to one aspect of the present application, a terahertz single-pixel acquisition system is provided, which is used to acquire an input image of a target, and to input the input image of the target into an image reconstruction network containing mask parameters for image reconstruction to obtain a reconstructed image of the target; the image reconstruction network is trained using training images in a training set, and the input image is acquired using multiple mask images obtained by the trained image reconstruction network.
[0015] According to one aspect of the present application, an electronic device includes at least one processor and at least one memory, wherein the memory stores program instructions or codes; the program instructions or codes are loaded and executed by the processor, so that the electronic device implements the real-time imaging method as described above.
[0016] According to one aspect of the present application, a storage medium stores program instructions or codes thereon, wherein the program instructions or codes are loaded and executed by a processor to implement the real-time imaging method as described above.
[0017] According to one aspect of the present application, an application product includes program instructions or codes, which are stored in a storage medium. A processor of an electronic device reads the program instructions or codes from the storage medium, loads and executes the program instructions or codes, so that the electronic device implements the real-time imaging method as described above.
[0018] The beneficial effects of the technical solution provided by this application are:
[0019] In the above technical solution, a training set is obtained; the training set includes multiple training images; an image reconstruction network containing mask parameters is trained using the training images in the training set, and multiple mask images are obtained based on the mask parameters in the trained image reconstruction network; the multiple mask images are preprocessed, and the preprocessed mask images are used to perform image acquisition in a terahertz single-pixel acquisition system to obtain an input image of the target; the input image of the target is input into the image reconstruction network for image reconstruction to obtain a reconstructed image of the target. In other words, by building an image reconstruction network through deep learning, and using mask parameters for training during image training, the design of a projection mask under undersampling conditions is achieved, and the image can be optimized, the image signal-to-noise ratio can be increased, and then the effect of reconstructing a high-quality image with a small number of masks can be achieved, thereby effectively solving the problem of poor imaging quality when the number of masks is small in the related art. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments of the present application.
[0021] FIG1 is a schematic diagram of an implementation environment according to the present application;
[0022] FIG2 is a flow chart showing a real-time imaging method according to an exemplary embodiment;
[0023] FIG3 is a flow chart of step 230 in one embodiment of the embodiment corresponding to FIG2 ;
[0024] FIG4 is a flow chart of step 250 in one embodiment of the embodiment corresponding to FIG2 ;
[0025] FIG5 is a schematic structural diagram of a terahertz single-pixel acquisition system according to an exemplary embodiment;
[0026] FIG6 is a flow chart of step 270 in one embodiment of the embodiment corresponding to FIG2 ;
[0027] FIG7 is a flow chart of step 275 in one embodiment of the embodiment corresponding to FIG6 ;
[0028] FIG8 is a schematic diagram of a specific implementation of a network structure of an image reconstruction network in an application scenario;
[0029] FIG9 a is a diagram showing an image reconstruction effect of a real-time imaging method in an application scenario;
[0030] FIG9 b is a diagram showing an image reconstruction effect of a real-time imaging method in an application scenario;
[0031] FIG10 is a structural block diagram of a real-time imaging device according to an exemplary embodiment;
[0032] FIG11 is a hardware structure diagram of a server according to an exemplary embodiment;
[0033] Fig. 12 is a structural block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION
[0034] The following describes embodiments of the present application in detail, with examples of the embodiments illustrated in the accompanying drawings. Throughout, identical or similar reference numerals denote identical or similar elements or elements having identical or similar functions. The embodiments described below with reference to the accompanying drawings are illustrative and intended solely to explain the present application and are not to be construed as limiting the present application. Those skilled in the art will appreciate that, unless otherwise specified, the singular forms "a," "an," "the," and "the" as used herein may include the plural forms. It should be further understood that, as used in the specification of the present application, the term "comprising" refers to the presence of the recited features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when an element is referred to as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or intervening elements may be present. Furthermore, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" as used herein includes all or any combination of one or more of the associated listed items.
[0035] At present, terahertz single-pixel imaging technology is mainly divided into three categories: 1) using a detector with no spatial resolution to scan the target pixel by pixel; 2) using a terahertz camera system based on a thermal detector array for image acquisition; 3) terahertz single-pixel imaging based on a single-pixel detector, which uses M masks to illuminate the target and records the total transmitted light or reflected light of each mask to obtain a reconstructed image of the target.
[0036] However, the first type of system is complex and takes a long time to image; the second type of system has low temporal / spectral resolution and low sensitivity; in the third type of system, the quality of terahertz single-pixel imaging is positively correlated with the number of coding masks, and a sufficient number of masks are required to ensure imaging quality, which is not conducive to cost savings.
[0037] Currently, image reconstruction algorithms can be divided into three categories: 1) compressed sensing; 2) base scanning; and 3) deep learning algorithms. None of these algorithms can guarantee image quality while reducing the number of masks. Furthermore, data-driven deep learning algorithms, which use a training set for deep learning, are often only applicable to the targets in the training set and cannot guarantee the diversity of imaged targets. Compressed sensing and base scanning, on the other hand, either require long image reconstruction times or long sampling times, and the high-frequency details in the reconstructed image are also unsatisfactory.
[0038] This shows that the related art still has the defect of poor imaging quality when the number of masks is small.
[0039] To this end, the real-time imaging method provided in the present application can achieve the effect of reconstructing high-quality images with a small number of masks, effectively improving the imaging quality. Accordingly, the real-time imaging method is suitable for a real-time imaging device, which can be deployed in an electronic device. The electronic device can be a computer device configured with a von Neumann architecture. For example, the computer device includes but is not limited to a desktop computer, a laptop computer, a server, etc.
[0040] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.
[0041] Figure 1 is a schematic diagram of an implementation environment involved in an image processing method. It should be noted that this implementation environment is only an example adapted to the present invention and should not be considered as providing any limitation on the scope of application of the present invention.
[0042] The implementation environment includes a collection end 110 and a service end 130 .
[0043] Specifically, the acquisition end 110 can also be considered as a terahertz single-pixel image acquisition device, including but not limited to a terahertz camera, a terahertz camera, and other electronic devices with terahertz single-pixel image acquisition capabilities. For example, the acquisition end 110 is a terahertz single-pixel acquisition system.
[0044] Server 130 can be an electronic device such as a desktop computer, laptop computer, or server. It can also be a computer cluster consisting of multiple servers, or even a cloud computing center consisting of multiple servers. Server 130 is used to provide background services, such as, but not limited to, image reconstruction services.
[0045] A network communication connection is pre-established between the server 130 and the acquisition terminal 110 via a wired or wireless method, and data transmission between the server 130 and the acquisition terminal 110 is achieved via the network communication connection. The transmitted data includes but is not limited to: the input image of the target, etc.
[0046] In one application scenario, through the interaction between the acquisition terminal 110 and the server 130 , the acquisition terminal 110 captures and acquires an input image for a target, and uploads the input image to the server 130 to request the server 130 to provide an image reconstruction service.
[0047] For the server 130, after receiving the input image of the target uploaded by the acquisition terminal 110, it calls the image reconstruction service and uses the image reconstruction network containing the mask parameters to reconstruct the input image of the target to obtain a reconstructed image of the target.
[0048] Among them, the image reconstruction network is trained using the training images in the training set, and the multiple mask images obtained by the trained image reconstruction network are used to collect the input image of the target at the collection end 110, so as to solve the problem of poor imaging quality when the number of masks is small in the related art.
[0049] Please refer to FIG. 2 . An embodiment of the present application provides a real-time imaging method. The method is applicable to an electronic device, which may be the server 130 in the implementation environment shown in FIG. 1 .
[0050] In the following method embodiments, for ease of description, the execution subject of each step of the method is taken as an electronic device as an example for illustration, but this does not constitute a specific limitation.
[0051] As shown in FIG2 , the method may include the following steps:
[0052] Step 210: Obtain a training set.
[0053] The training set includes multiple training images.
[0054] In one possible implementation, the MINIST dataset of a set resolution is split into two datasets at a ratio of 9:1. The larger dataset is used as the training set for subsequent training of the image reconstruction network, and the smaller dataset is used as the test set / validation set to evaluate the training effect of the image reconstruction network after training. The set resolution can be flexibly adjusted according to the actual needs of the application scenario and is not limited here. For example, in the application scenario of high-resolution image reconstruction, the set resolution is 32×32.
[0055] Step 230 : Using the training images in the training set to train the image reconstruction network including the mask parameters, and obtaining a plurality of mask images based on the mask parameters in the trained image reconstruction network.
[0056] Among them, the image reconstruction network is a deep learning network for image reconstruction and outputting mask images obtained by training using training images in the training set.
[0057] In one possible implementation, the image reconstruction network structure includes an encoding layer that simulates single-pixel sampling through encoding, a decoding layer for singular value decomposition, and an enhancement layer for performing enhancement processing. In this approach, the encoding layer can simulate the single-pixel sampling process to encode the input image and obtain a mask image. The decoding layer can perform singular value decomposition on the image to increase the diversity of imaging targets. The enhancement layer can perform image enhancement processing, improving the quality of the reconstructed image by the image reconstruction network.
[0058] In one possible implementation, the enhancement layer includes multiple downsampling layers, multiple upsampling layers, and multiple residual connection layers connected between the downsampling and upsampling layers. The downsampling layers are used for feature extraction, the upsampling layers are used for feature fusion, and the residual connection layers are used for feature propagation between the connected downsampling and upsampling layers. In this way, the enhancement layer is implemented through downsampling and upsampling, which can effectively improve the quality of the reconstructed image during the image reconstruction process and make the reconstructed image of the target closer to the real image.
[0059] As shown in FIG3 , the training process of the image reconstruction network in step 230 may include the following steps:
[0060] Step 231: Input a current training image in the training set into the image reconstruction network for training to obtain an output image corresponding to the current training image.
[0061] That is to say, during the training process, by inputting the training image into the image reconstruction network, an output image corresponding to the training image can be obtained, so as to judge whether the image reconstruction network has completed the training by comparing the structural similarity between the output image and the training image.
[0062] Step 233: Calculate the loss value based on the difference between the current training image and its corresponding output image.
[0063] Among them, each loss value is determined based on the structural similarity between each output image and the corresponding training image. When the loss value reaches the minimum value, it can be regarded as the loss value meets the convergence condition.
[0064] In one possible implementation, during the training process of the image reconstruction network, the loss value is obtained by calculating the Euclidean distance between the training image and the output image, and then the image reconstruction network is supervised trained based on the Euclidean distance using the Adam optimizer to obtain a trained image reconstruction network that has the ability to reconstruct the target input image.
[0065] It is worth mentioning that the structural similarity between the output image and the corresponding training image is not limited to being determined by Euclidean distance, but can also be determined by similarity calculation methods such as Ming distance, Manhattan distance, Chebyshev distance, cosine similarity, Pearson correlation coefficient, and Jaccard similarity coefficient, which does not constitute a specific limitation here.
[0066] In step 235, if the loss value meets the convergence condition, the training is completed and the image reconstruction network is obtained; otherwise, the parameters of the image reconstruction network are updated, and the next training image in the training set is obtained to continue training the image reconstruction network until the loss value meets the convergence condition.
[0067] Among them, the parameters of the image reconstruction network include mask parameters.
[0068] Specifically, after determining that the loss value does not meet the convergence conditions, backpropagation can be used to update the parameters corresponding to each layer in the image reconstruction network. The updated parameters include but are not limited to the mask parameters. The mask parameters, by participating in the encoding process of the encoding layer, can affect the image reconstruction effect of the image reconstruction model. Adjusting the mask parameters during training can train the image reconstruction model in a direction that improves image reconstruction capabilities.
[0069] After the image reconstruction network adjusts its parameters, the next training image is input into the image reconstruction network with adjusted parameters for training. The image reconstruction network is trained iteratively until the loss value meets the convergence condition. The image reconstruction network is considered to have reached a convergence state, and the training is completed. A trained image reconstruction network with the ability to reconstruct the target input image is obtained.
[0070] In the above process, deep learning methods were applied to further adjust the effective image reconstruction network, so that the image reconstruction network can generate high-quality reconstructed images, improve the image reconstruction effect, ensure the applicability of the image reconstruction network, improve the image reconstruction quality, integrate model driving into data-driven deep learning, and improve the robustness of the system.
[0071] During the training process of the image reconstruction network, the mask image participates in the training of the image reconstruction network as the convolution kernel of the coding layer convolution operation. The mask parameters corresponding to the mask image are adjusted during the training process. After the training of the image reconstruction network is completed, the optimal mask parameters are obtained through the coding layer to obtain the corresponding mask image, so that the target input image can be acquired in the terahertz single-pixel acquisition system in the future.
[0072] Step 250 , pre-processing the multiple mask images, and using the pre-processed mask images to perform image acquisition in a terahertz single-pixel acquisition system to obtain an input image of the target.
[0073] Among them, the terahertz single-pixel acquisition system is a system that uses terahertz waves to project a mask image onto a target to collect information about the target. It is used to collect the input image of the target, input the input image of the target into an image reconstruction network containing mask parameters for image reconstruction, and obtain a reconstructed image of the target.
[0074] As shown in FIG4 , the preprocessing process of the mask image in step 250 may include the following steps:
[0075] Step 251 : interpolating a plurality of mask images according to a set image size, and modulating the interpolated mask images to obtain a pre-processed mask image.
[0076] Interpolation refers to predicting the value of a pixel in the mask image based on information about the surrounding pixels using an interpolation algorithm, thereby changing the resolution of the mask image.
[0077] Modulation refers to the process of controlling the terahertz irradiation target through the semiconductor photocurrent effect to generate a modulation signal that meets the modulation requirements.
[0078] Step 253 : In the terahertz single-pixel acquisition system, the pre-processed mask image is projected onto the target using the terahertz wave, so that the target generates a modulation signal.
[0079] Specifically, as shown in Figure 5, the terahertz single-pixel acquisition system includes a digital micromirror device for recording the mask image, a femtosecond laser for generating laser light, a silicon wafer for carrying the target, and a single-pixel detector for receiving the modulated signal.
[0080] In a terahertz single-pixel acquisition system, after acquiring a mask image, the mask image is burned into a digital micromirror device (DMD). An 808nm semiconductor laser is then used to illuminate the DMD. The laser then passes through a lens, reflecting the mask image onto the silicon wafer in the target area. The terahertz wave is then modulated using the semiconductor photocurrent effect, causing the terahertz wave to modulate the target. The modulated terahertz wave then outputs a modulated signal on a single-pixel detector.
[0081] Step 255: Obtain the target input image by collecting the modulated signal.
[0082] In one possible implementation, the acquisition process uses a femtosecond laser to illuminate the terahertz transmitter through a beam splitter to generate terahertz waves for acquisition. The terahertz waves pass through two off-axis parabolic mirrors and a target tightly attached to a high-resistance silicon wafer to reach the terahertz receiver. During this process, the terahertz waves carry the modulation information generated by the modulation of the target. After reaching the terahertz receiver, the modulation information is restored to a 32×32 resolution input image and input into the image reconstruction network for image reconstruction.
[0083] Under the influence of the above-mentioned embodiment, the terahertz single-pixel acquisition system projects the target based on the mask image to collect information, realizes the acquisition of the input image using the mask image, improves the efficiency of the terahertz single-pixel acquisition system, and improves the imaging quality. Moreover, since the obtained input image resolution is 32×32, which is much larger than the number of masks used in the coding layer, it can improve the image reconstruction efficiency, accelerate the image reconstruction speed, and realize terahertz single-pixel rapid imaging.
[0084] Step 270 , reconstructing the input image of the target using a reconstruction network to obtain a reconstructed image of the target.
[0085] Specifically, as shown in FIG6 , step 270 may include the following steps:
[0086] Step 271: Input the target input image into the coding layer, and use the mask parameters in the coding layer to simulate single-pixel sampling of the input image to obtain a coding result.
[0087] Specifically, the single-pixel acquisition process is simulated by using the encoding layer to perform convolution operations on the training image and mask parameters. The specific process is given by the formula Determine that the sampling formula for the M mask image sampling process is S M×1 =P M×L O L×1 , where p i is the mask parameter, which is an N×N matrix. The image resolution is N×N, u and v represent the pixel coordinates respectively, and S M×1 It is made by i The column vector, P M×L Each row of is a row vector corresponding to the mask image representation, O L×1 is a column vector representing the training images.
[0088] Step 273: Input the encoding result into the decoding layer for singular value decomposition to obtain a decoding result.
[0089] After obtaining the encoding result output by the encoding layer, the decoder performs singular value decomposition on the encoding result. The singular value decomposition process is as follows: Where L = N × N, P M×L is a matrix containing M masks, p i That is, P M×L The i-th row of the matrix is the i-th mask among the M masks.
[0090] The decoding result output by the singular value decomposition of the decoding layer is O L×1 =(VV T ) -1 VΣ -1 (U T U)-1 U T S, where O L×1 It is the result of singular value decomposition, which is an L×1 vector. It is compressed into an N×N matrix, which is the decoding result output by the decoding layer.
[0091] In the above process, using singular value decomposition to decode the encoded results can avoid problems such as poor generalization caused by using deep learning methods in the image reconstruction process, and prevent the image reconstruction network from only being able to effectively reconstruct the training images in the training set. This can improve the generalization of the image reconstruction network and ensure the diversity of image reconstruction targets.
[0092] Step 275: Input the decoding result into the enhancement layer for enhancement processing to obtain the target reconstructed image.
[0093] After obtaining the decoding result output by the decoding layer, the decoding result is input into the enhancement layer for enhancement processing to obtain a reconstructed image.
[0094] In this embodiment, the enhancement layer includes multiple downsampling layers, multiple upsampling layers, and multiple residual connection layers respectively connected between the downsampling layers and the upsampling layers.
[0095] Specifically, as shown in FIG7 , step 275 may include the following steps:
[0096] Step 2751, perform feature extraction on the decoding result through multiple downsampling layers to obtain multiple input features.
[0097] Step 2753: Feature propagation is performed between the connected upsampling layer and downsampling layer through multiple residual connection layers, and multiple input features are transferred from the downsampling layer to the corresponding upsampling layer.
[0098] In step 2755, multiple input features are fused through multiple upsampling layers to obtain a reconstructed image of the target.
[0099] At the first downsampling layer, features are extracted through several convolutional blocks to obtain the feature map of this downsampling layer. This feature map is then transmitted as input features to the first upsampling layer through feature propagation via the residual connection layer. Furthermore, the feature map of the first downsampling layer is transmitted to the second downsampling layer, where it is extracted through several convolutional blocks to form the input features of the second downsampling layer. This process continues in this manner until all downsampling layers have completed feature extraction.
[0100] Among them, the purpose of the downsampling layer is to scale-compress the input feature map. Accordingly, the number of convolution blocks in different downsampling layers is different, so that feature maps with different numbers of channels can be obtained, achieving the effect of scale compression.
[0101] For the last downsampling layer, the input features are transferred to the last upsampling layer for feature fusion through multiple upsampling layers.
[0102] In the last upsampling layer, the feature map that has completed feature fusion will be transmitted to the last upsampling layer to continue feature fusion until all upsampling layers complete the feature fusion process, and the reconstructed image of the target is obtained.
[0103] In this process, the image reconstruction network combines residual dense connections with downsampling. Each compression step reduces the network depth of the corresponding layer, improving image reconstruction efficiency. Furthermore, the downsampling process ensures that valid information is obtained after each downsampling training, thereby improving the quality of the reconstructed image and making it closer to the real image, thus ensuring the quality of the reconstructed image.
[0104] FIG8 and FIG9a to FIG9b are schematic diagrams showing a specific implementation of a real-time imaging method in an application scenario.
[0105] In this application scenario, the image reconstruction network is trained using the MINIST dataset with a resolution of 32×32. After the training, the model parameters including the mask parameters are fixed, and the trained image reconstruction network is encoded into the output mask image.
[0106] Refer back to Figure 5. The terahertz single-pixel acquisition system includes a digital micromirror device (DMD), a femtosecond laser, and a single-pixel detector, including a femtosecond laser, an off-axis parabolic mirror, a lens, a terahertz transmitter, a terahertz receiver, and a silicon chip. The terahertz single-pixel acquisition system captures the input image of a target. Specifically, the trained 32×32 mask image is interpolated to 768×768 and converted to a binary image of 0s and 1s before being burned into the DMD. The DMD flip speed is set to 10 kHz, the acquisition card sampling rate is 2 MHz, and the synchronization signal is 10 μs. A photodiode is then used as a terahertz modulator to emit a sinusoidal fringe-coded terahertz wave. This is achieved by illuminating the DMD with a femtosecond laser, triggering it to project the mask image onto the target. During the triggering of the DMD, the sampling duration is set to 10 μs, and the number of projected mask images is compressed to 25% using the l1-norm algorithm. Then, the femtosecond laser in the single-pixel detector is used to illuminate the terahertz transmitting end to emit terahertz waves, and the off-axis parabolic mirror and beam splitter are used to adjust the terahertz wave optical path so that the terahertz wave passes through the target attached to the silicon wafer and reaches the terahertz receiving end. Finally, the input image of the target is collected by the single-pixel detector.
[0107] Figure 8 shows a schematic diagram of the specific implementation of the network structure of the image reconstruction network in an application scenario. In Figure 8, the image reconstruction network includes an encoding layer, a decoding layer, and an enhancement layer. Among them, the enhancement layer includes multiple downsampling layers, multiple upsampling layers, and residual connection layers connected between the upsampling layer and the downsampling layer.
[0108] The input image of the target collected by the terahertz single-pixel acquisition system is used as the input of the image reconstruction network. Through the image reconstruction of the input image by the image reconstruction network, a reconstructed image with an image resolution of 32×32 is reconstructed.
[0109] During the image reconstruction process, the number of masks M = 128, and 36 of them are randomly selected for display, as shown on the left side of Figure 9a. The reconstructed images obtained by randomly selecting 64 input images in the test set as the network input are shown on the right side of Figure 9a. Figure 9b shows a schematic diagram of the image reconstruction effects of three different types of targets. These three different types of targets include the number "3", the letter "H", and two Chinese characters "千" and "手". Among them, the letter and Chinese characters do not appear in the training set.
[0110] During the entire test process, SSIM = 0.98828, PSNR = 32.0005, which indicates that the generalization of the image reconstruction network is good. In addition, the time taken by the terahertz single-pixel acquisition system to collect 128 masks is about 10 ms, so fast and even real-time imaging can be achieved.
[0111] In this application scenario, fast imaging is achieved by outputting the mask image through the image reconstruction network. Singular value decomposition is used in the image reconstruction network to ensure the diversity of imaging targets, and deep learning is used to achieve high-quality image reconstruction effects, which can ensure the diversity of imaging targets and improve the imaging quality with a small number of masks.
[0112] The following is an apparatus embodiment of the present application, which can be used to execute the real-time imaging method involved in the present application. For the details not disclosed in the apparatus embodiment of the present application, please refer to the method embodiment of the real-time imaging method involved in the present application.
[0113] Please refer to Figure 10. In the embodiment of the present application, a real-time imaging device 1000 is provided, including but not limited to: a training set acquisition module 1010, a training module 1030, a preprocessing module 1050, and an image reconstruction module 1070.
[0114] Among them, the training set acquisition module 1010 is used to acquire a training set; the training set includes multiple training images.
[0115] The training module 1030 is configured to train an image reconstruction network including mask parameters using training images in a training set, and obtain a plurality of mask images based on the mask parameters in the trained image reconstruction network.
[0116] The preprocessing module 1050 is used to preprocess the multiple mask images, and use the preprocessed mask images to perform image acquisition in the terahertz single-pixel acquisition system to obtain an input image of the target.
[0117] The image reconstruction module 1070 is used to reconstruct the input image of the target through the reconstruction network to obtain a reconstructed image of the target.
[0118] It should be noted that the real-time imaging device provided in the above embodiment is only illustrated by the division of the above functional modules when performing real-time imaging. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the real-time imaging device will be divided into different functional modules to complete all or part of the functions described above.
[0119] In addition, the real-time imaging device and the real-time imaging method provided in the above embodiments belong to the same concept, wherein the specific manner in which each module performs operations has been described in detail in the method embodiments and will not be repeated here.
[0120] Fig. 11 shows a schematic diagram of the structure of a server according to an exemplary embodiment. The server is applicable to the server 130 in the implementation environment shown in Fig. 1 .
[0121] It should be noted that the server is only an example adapted for the present application and cannot be considered to provide any limitation on the scope of use of the present application. The server cannot be interpreted as needing to rely on or necessarily having one or more components in the exemplary server 2000 shown in Figure 11.
[0122] The hardware structure of the server 2000 may vary greatly due to different configurations or performances. As shown in FIG11 , the server 2000 includes a power supply 210 , an interface 230 , at least one memory 250 , and at least one central processing unit (CPU) 270 .
[0123] Specifically, the power supply 210 is used to provide operating voltage for each hardware device on the server 2000 .
[0124] The interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices, such as the interaction between the terminal 100 and the server 200 in the implementation environment shown in FIG1 .
[0125] Of course, in other examples adapted by this application, the interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input-output interface 235, and at least one USB interface 237, as shown in FIG11 , which is not specifically limited here.
[0126] The memory 250 serves as a carrier for resource storage and can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon include an operating system 251, application 253 and data 255, etc. The storage method can be temporary storage or permanent storage.
[0127] Among them, the operating system 251 is used to manage and control the various hardware devices and application programs 253 on the server 2000 to enable the central processing unit 270 to calculate and process the massive data 255 in the memory 250. It can be Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.
[0128] The application 253 is a program instruction or code that performs at least one specific task based on the operating system 251. It may include at least one module (not shown in FIG. 11 ), each of which may include program instructions or code for the server 2000. For example, a real-time imaging device may be considered an application 253 deployed on the server 2000.
[0129] The data 255 may be a photo or picture stored in a disk, or an input image of a target, etc., stored in the memory 250 .
[0130] The central processing unit 270 may include one or more processors and is configured to communicate with the memory 250 via at least one communication bus to read program instructions or codes stored in the memory 250, thereby performing operations and processing on the massive amount of data 255 in the memory 250. For example, the real-time imaging method can be implemented by the central processing unit 270 reading a series of program instructions or codes stored in the memory 250.
[0131] In addition, the present application can also be implemented through hardware circuits or hardware circuits combined with software. Therefore, the implementation of the present application is not limited to any specific hardware circuits, software, or a combination of the two.
[0132] Please refer to FIG. 12 . An electronic device 4000 is provided in an embodiment of the present application. The electronic device 4000 may include a desktop computer, a laptop computer, a server, etc.
[0133] In FIG. 12 , the electronic device 4000 includes at least one processor 4001 and at least one memory 4003 .
[0134] Data exchange between the processor 4001 and the memory 4003 can be achieved through at least one communication bus 4002. The communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in FIG12, but this does not mean that there is only one bus or one type of bus.
[0135] Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data exchange between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the number of transceivers 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.
[0136] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.
[0137] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program instructions or codes in the form of instructions or data structures and can be accessed by the electronic device 400, but is not limited to these.
[0138] Program instructions or codes are stored in the memory 4003 , and the processor 4001 can read the program instructions or codes stored in the memory 4003 through the communication bus 4002 .
[0139] When the program instructions or codes are executed by the processor 4001 , the real-time imaging methods in the above-mentioned embodiments are implemented.
[0140] In addition, an embodiment of the present application provides a storage medium on which program instructions or codes are stored. The program instructions or codes are loaded and executed by a processor to implement the above-mentioned real-time imaging method.
[0141] In an embodiment of the present application, an application product is provided, which includes program instructions or codes, and the program instructions or codes are stored in a storage medium. The processor of the electronic device reads the program instructions or codes from the storage medium, loads and executes the program instructions or codes, so that the electronic device implements the real-time imaging method as described above.
[0142] Compared with related technologies, the present application achieves fast imaging by outputting mask images through an image reconstruction network, uses singular value decomposition in the image reconstruction network to ensure the diversity of imaging targets, and uses deep learning to achieve high-quality image reconstruction effects. It can ensure the diversity of imaging targets with a small number of masks and improve imaging quality.
[0143] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
[0144] The above description is only part of the implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A real-time imaging method, It is characterized in that The method comprises: Acquire a training set; the training set includes a plurality of training images; Using the training images in the training set to train an image reconstruction network including mask parameters, and obtaining a plurality of mask images based on the mask parameters in the trained image reconstruction network; Preprocessing a plurality of mask images, and using the preprocessed mask images to perform image acquisition in a terahertz single-pixel acquisition system to obtain an input image of the target; The input image of the target is input into the image reconstruction network for image reconstruction to obtain a reconstructed image of the target.
2. The method according to claim 1, It is characterized in that The step of training the image reconstruction network including mask parameters using the training images in the training set includes: Inputting a current training image in the training set into the image reconstruction network for training to obtain an output image corresponding to the current training image; The loss value is calculated based on the difference between the current training image and its corresponding output image; If the loss value satisfies the convergence condition, the training is completed and the image reconstruction network is obtained; Otherwise, update the parameters of the image reconstruction network, and obtain the next training image in the training set to continue training the image reconstruction network until the loss value meets the convergence condition; the parameters of the image reconstruction network include the mask parameters.
3. The method according to claim 1, It is characterized in that The image reconstruction network includes a coding layer, a decoding layer, and an enhancement layer; The step of inputting the input image of the target into the image reconstruction network for image reconstruction to obtain a reconstructed image of the target comprises: Inputting the input image of the target into the coding layer, and using the mask parameters in the coding layer to simulate single-pixel sampling of the input image to obtain a coding result; Inputting the encoding result into the decoding layer for singular value decomposition to obtain a decoding result; The decoding result is input into the enhancement layer for enhancement processing to obtain a reconstructed image of the target.
4. The method according to claim 3, It is characterized in that The enhancement layer includes a plurality of down-sampling layers, a plurality of up-sampling layers, and a plurality of residual connection layers respectively connected between the down-sampling layers and the up-sampling layers; The step of inputting the decoding result into the enhancement layer for enhancement processing to obtain a reconstructed image of the target includes: Performing feature extraction on the decoding result through multiple downsampling layers to obtain multiple input features; Through multiple residual connection layers, feature propagation is performed between the connected upsampling layer and the downsampling layer, and multiple input features are respectively transmitted from the downsampling layer to the corresponding upsampling layer; Through multiple upsampling layers, multiple input features are fused to obtain a reconstructed image of the target.
5. The method according to claim 1, It is characterized in that The preprocessing of the multiple mask images comprises: A plurality of mask images are interpolated according to a set image size, and the interpolated mask images are modulated to obtain a preprocessed mask image.
6. The method according to any one of claims 1 to 5, It is characterized in that The method of using the preprocessed mask image to perform image acquisition in a terahertz single-pixel acquisition system to obtain an input image of the target includes: In the terahertz single-pixel acquisition system, the pre-processed mask image is projected onto the target using terahertz waves, so that the target generates a modulation signal; By collecting the modulated signal, an input image of the target is obtained.
7. A terahertz single-pixel acquisition system, It is characterized in that The system is used to collect an input image of a target, and input the input image of the target into an image reconstruction network including mask parameters for image reconstruction, so as to obtain a reconstructed image of the target; The image reconstruction network is trained using training images in a training set, and the input image is collected using a plurality of mask images obtained by completing the trained image reconstruction network.
8. A real-time imaging device, It is characterized in that The device comprises: A training set acquisition module, used to acquire a training set; the training set includes a plurality of training images; A training module, used to train an image reconstruction network including mask parameters using the training images in the training set, and obtain a plurality of mask images based on the mask parameters in the trained image reconstruction network; An image acquisition module is used to preprocess a plurality of mask images, and use the preprocessed mask images to perform image acquisition in a terahertz single-pixel acquisition system to obtain an input image of the target; The image reconstruction module is used to input the input image of the target into the image reconstruction network for image reconstruction to obtain a reconstructed image of the target.
9. An electronic device, It is characterized in that include: at least one processor and at least one memory, wherein: The memory stores program instructions or codes; The program instructions or codes are loaded and executed by the processor, so that the electronic device implements the real-time imaging method according to any one of claims 1 to 6.
10. A storage medium having program instructions or codes stored thereon, It is characterized in that The program instructions or codes are loaded and executed by a processor to implement the real-time imaging method according to any one of claims 1 to 6.
Citation Information
Patent Citations
Near-field terahertz wave spectral imaging system and method
CN109444085A
Single-pixel terahertz wave imaging method and system
CN110231292A
Terahertz single-pixel imaging method and terahertz single-pixel imaging system
CN112525851A
Terahertz single-pixel imaging method and system
CN114894740A
Real-time imaging method and device, electronic equipment and storage medium
CN116740486A
Cited By
Steel bar signal suppression and reinforced concrete structure defect reconstruction method and system
CN122017778A
