Compression ultrafast imaging device based on instantaneous image reference

Abstract

The invention discloses a compression ultrafast imaging device based on instantaneous image reference. The compression ultrafast imaging device is composed of a dynamic scene acquisition system, an instantaneous imaging system, a coding compression imaging system and a control and processing system. According to the method, dynamic scenes including coding compression imaging and instantaneous imaging are recorded simultaneously through two different imaging models, and collected bimodal data are processed and reconstructed by using a bimodal information fusion image reconstruction algorithm based on a non-training neural network, so that a high-fidelity dynamic scene is recovered. According to the method, the image quality of compression ultrafast imaging can be effectively improved, the defect of a traditional compression ultrafast imaging technology in the aspect of image fidelity is overcome, the temporal-spatial resolution of an imaging system is remarkably improved, and the method is suitable for fine observation of ultrafast dynamic phenomena and particularly has important application value in the fields of scientific research, industrial application and military affairs.

Description

Technical Field

[0001] The present invention relates to the field of optical ultrafast imaging technology, and in particular to a compressed ultrafast imaging device based on instantaneous image reference. Background Art

[0002] Single-exposure ultrafast optical imaging can capture the continuous spatio-temporal information of an instantaneous scene within a single exposure, and has become an indispensable tool for observing non-repeatable or destructive ultrafast phenomena. Due to its crucial significance in scientific research, industrial applications, and the military field, single-exposure ultrafast optical imaging technologies based on various strategies have been developed in recent years. Generally, they can be divided into two categories according to whether an illumination source is required, namely active illumination-based ultrafast optical imaging and passive reception-based ultrafast optical imaging. Active illumination-based ultrafast optical imaging detects ultrafast dynamics by transforming time information into other domains, such as spectral, spatial, spatial frequency, and polarization domains. Representative examples include frequency identification algorithms with multiple exposures, chirped spectral mapping ultrafast imaging, sequential timing all-optical mapping imaging, and polarization-resolved ultrafast mapping imaging. Active illumination-based ultrafast optical imaging usually involves the modulation of illumination light, which results in a complex imaging system and limits the applicability of detecting self-emitting ultrafast events. Passive reception-based ultrafast optical imaging can directly record dynamic scenes through an imaging system with ultrafast time resolution, such as high-speed sampling cameras, ultrafast framing cameras, and compressed ultrafast imaging. Among these technologies, compressed ultrafast imaging has received extensive attention due to its high frame rate and large sequence depth, and has been applied to many fields, such as light propagation, fluorescence imaging, plasma dynamics, ultrafast optical springs, and optical chaos.

[0003] Compressive ultrafast imaging combines the concepts of compressive sensing and streak imaging. In compressive ultrafast imaging, a three-dimensional spatio-temporal dynamic scene is spatially encoded, temporally sheared, and spatio-temporally integrated into a two-dimensional compressed image, which is then reconstructed from the compressed image by an algorithm based on compressive sensing. Compressive ultrafast imaging excels in imaging speed and sequence depth, but due to lossy coding and spatio-temporal mixing during the image acquisition process, its image quality is inferior to other ultrafast imaging techniques. This has been a key issue in compressive ultrafast imaging and has hindered further applications with high spatio-temporal resolution. To address this problem, various methods have been developed by previous researchers to improve the imaging quality of compressive ultrafast imaging from both the hardware and algorithm aspects. Some methods aim to increase information sampling by improving the hardware system. For example, based on the bidirectional characteristics of digital micromirror devices, a complementary coding strategy is adopted to double the information collected in compressive ultrafast imaging. Similarly, a multi-channel sampling strategy has also been employed in compressive ultrafast imaging to further improve the overall sampling rate. Meanwhile, the optimal coding strategy also helps to improve the imaging quality of compressive ultrafast imaging because it reduces the correlation between spatial coding and the dynamic scene, thus improving the sampling efficiency. In addition, the camera that records the spatio-temporal integrated image of the dynamic scene provides additional spatial and intensity constraints for image reconstruction in compressive ultrafast imaging, resulting in better image quality. In some other methods, image reconstruction is promoted by choosing a better iterative framework or more prior knowledge. For example, an enhanced Lagrangian framework is used to optimize the iterative process in compressive ultrafast imaging, thus avoiding the selection of penalty parameters during image reconstruction. Plug-and-play frameworks based on generalized alternating projection or alternating direction method of multipliers are also used, which have a high flexibility in combining various priors. To obtain better performance, total variation minimization and non-local similarity can be used to constrain image reconstruction. In addition, deep learning-based priors, such as FFDNET, DRUNET, and FASTDVDNET, are also embedded to improve the image quality. Although a large amount of work on hardware and reconstruction algorithms has been reported to improve the image quality of compressive ultrafast imaging, its performance still cannot meet the requirements for observing fine ultrafast dynamics. Summary of the Invention

[0004] The object of the present invention is to provide a compressive ultrafast imaging device based on instantaneous image reference for the insufficient fidelity of compressive ultrafast imaging. By using two different systems to record the same dynamic scene, including a compressive ultrafast imaging system and an instantaneous imaging system, and a non-trained neural network as a generator to recover three-dimensional spatio-temporal data, combining the instantaneous reference information and spatial-intensity constraints, the image quality that cannot be achieved by conventional compressive ultrafast imaging is achieved.

[0005] The specific technical solution for achieving the object of the present invention is as follows:

[0006] A compressive ultrafast imaging device based on instantaneous image reference, which comprises:

[0007] A dynamic scene acquisition system consisting of an objective lens and a dynamic scene;

[0008] The objective lens of the dynamic scene acquisition system and the dynamic scene are connected in optical path in sequence;

[0009] An instantaneous imaging system consisting of an instantaneous camera, a first convex lens and a beam splitter;

[0010] The instantaneous camera, the first convex lens and the beam splitter of the instantaneous imaging system are connected in optical path in sequence for the reflected light path;

[0011] An encoded compressive imaging system consisting of a streak camera, a second convex lens, a third convex lens, a chromium glass mask and a fourth convex lens;

[0012] The streak camera, the second convex lens, the third convex lens, the chromium glass mask and the fourth convex lens of the encoded compressive imaging system are connected in optical path in sequence;

[0013] A control and processing system consisting of a digital delay pulse generator and a computer;

[0014] The digital delay pulse generator of the control and processing system is connected to the computer by a data line;

[0015] The objective lens of the dynamic scene acquisition system is connected in optical path with the beam splitter of the instantaneous imaging system;

[0016] The transmitted light path of the beam splitter of the instantaneous imaging system is connected in optical path with the fourth concave lens of the encoded compressive imaging system;

[0017] The computer of the control and processing system is respectively connected to the instantaneous camera of the instantaneous imaging system and the streak camera of the encoded compressive imaging system by data lines;

[0018] The digital delay pulse generator of the control and processing system is respectively connected to the instantaneous camera of the instantaneous imaging system and the streak camera of the encoded compressive imaging system by data lines;

[0019] The digital pulse generator of the control and processing system controls the input trigger signals of the instantaneous camera and the streak camera to ensure that each imaging system simultaneously acquires the dynamic scene in the same time period; finally, the computer processes and reconstructs the imaging results;

[0020] The reconstruction adopts a dual-modal information fusion image reconstruction algorithm based on an untrained neural network, and the high-fidelity dynamic scene can be restored by integrating the dual-modal information through this algorithm;

[0021] The present invention includes optical sampling and image reconstruction algorithms; the dynamic scene system performs optical sampling, and simultaneously records the dynamic scene through an instantaneous imaging system and a coded compressive imaging system, including coded compressive imaging and instantaneous imaging. The image reconstruction algorithm, namely the dual-modal information fusion image reconstruction algorithm based on an untrained neural network, is responsible for restoring a high-fidelity dynamic scene by integrating dual-modal information.

[0022] In optical sampling, and are the measurement images of coded compressive imaging and instantaneous imaging respectively, represents the dynamic scene to be measured, represents the time-shearing operator, represents the spatial encoding operator, represents the instantaneous sampling operator. The image acquisition process can be simply expressed by the following formula as

[0023]

[0024]

[0025] The image reconstruction algorithm of the present invention is the dual-modal information fusion image reconstruction algorithm based on an untrained neural network. By this algorithm, integrating dual-modal information can restore a high-fidelity dynamic scene. In image reconstruction, the instantaneous image is copied to have the same number as the reconstructed image, denoted as , and is used as the initial content to input into the neural network, and then the neural network with randomly initialized parameters processes the input image. The output three-dimensional spatio-temporal scene obtains the estimated measurement value through the imaging model. Subsequently, the estimated measurement value is compared with the experimental result, and the loss related to the TV constraint is calculated. Through this process, the image reconstruction is transformed into the optimization of the neural network parameters, and its formula is

[0026]

[0027] is the optimal parameter of the neural network corresponding to the final reconstructed image, , are the weights of the fidelity terms corresponding to coded compressive imaging and instantaneous imaging respectively, is the weight of the TV constraint term. represents the L2 norm, represents the TV norm. The loss is calculated for each iteration, and the model parameters are updated using the alternating direction method of multipliers optimizer. After a certain number of iterations, the neural network can reconstruct a high-quality three-dimensional spatio-temporal scene .

[0028] The present invention adopts a dual - modal information fusion image reconstruction algorithm based on a non - trained neural network. Through this algorithm, different acquired imaging results can be reconstructed into a high - quality three - dimensional spatio - temporal scene.

[0029] The present invention has the following technical advantages:

[0030] The present invention simultaneously records dynamic scenes using two different imaging modes, including coded compressive imaging and instantaneous imaging, achieving high - fidelity single - exposure ultrafast imaging. Compared with traditional compressive ultrafast imaging techniques, the present invention adopts a dual - modal information fusion image reconstruction algorithm based on a non - trained neural network, thereby obtaining higher fidelity in both the spatial and temporal domains. Specifically, coded compressive imaging records the spatio - temporal integrated information of the dynamic scene, and instantaneous imaging provides spatial information references for several instantaneous frames. By fusing the information of these two imaging modes, the present invention significantly improves the quality and accuracy of the reconstructed image during the image reconstruction process. Both simulation and experimental results show that the present invention can effectively enhance the fidelity of the reconstructed image, providing a powerful tool for capturing ultrafast dynamic phenomena with fine details. Compared with traditional compressive ultrafast imaging, the present invention has significant improvements in both imaging quality and speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is the structural diagram of the present invention;

[0032] Figure 2 is the schematic diagram of the imaging process of the present invention;

[0033] Figure 3 is the schematic diagram of the dual - modal information fusion image reconstruction algorithm based on a non - trained neural network of the present invention;

[0034] Figure 4 is the schematic diagram of the specific device for nanosecond - scale dynamic scene imaging in the embodiment of the present invention;

[0035] Figure 5 is the imaging result diagram of the nanosecond - scale dynamic scene in the embodiment of the present invention;

[0036] Figure 1 In the figure: 100 - dynamic scene acquisition system; 101 - dynamic scene; 102 - objective lens; 200 - instantaneous imaging system; 201 - beam splitter; 202 - first convex lens; 203 - instantaneous camera; 300 - coded compressive imaging system; 301 - fourth convex lens; 302 - chromium glass mask; 303 - third convex lens; 304 - second convex lens; 305 - streak camera; 400 - control and processing system; 401 - digital delay pulse generator; 402 - computer. DETAILED DESCRIPTION OF THE INVENTION

[0037] The present invention will be described in detail below in conjunction with the accompanying drawings and embodiments.

[0038] Referring to Figure 1 , the present invention is a compressed ultrafast imaging device based on instantaneous image reference, which includes: a dynamic scene acquisition system 100 composed of an objective lens 102 and a dynamic scene 101.

[0039] The objective lens 102 and the dynamic scene 101 of the dynamic scene acquisition system 100 are connected in optical path in sequence.

[0040] An instantaneous imaging system 200 composed of an instantaneous camera 203, a first convex lens 202 and a beam splitter 201.

[0041] The instantaneous camera 203, the first convex lens 202 and the beam splitter 201 of the instantaneous imaging system 200 are connected in optical path in sequence by reflection.

[0042] An encoding and compression imaging system 300 composed of a streak camera 305, a second convex lens 304, a third convex lens 303, a chromium glass mask 302 and a fourth convex lens 301.

[0043] The streak camera 305, the second convex lens 304, the third convex lens 303, the chromium glass mask 302 and the fourth convex lens 301 of the encoding and compression imaging system 300 are connected in optical path in sequence.

[0044] A control and processing system 400 composed of a digital delay pulse generator 401 and a computer 402;

[0045] The digital delay pulse generator 401 and the computer 402 of the control and processing system 400 are connected by data lines;

[0046] The objective lens 102 of the dynamic scene acquisition system 100 is connected in optical path with the beam splitter 201 of the instantaneous imaging system 200.

[0047] The transmitted light path of the beam splitter 201 of the instantaneous imaging system 200 is connected in optical path with the fourth concave lens 301 of the encoding and compression imaging system 300;

[0048] The computer 402 of the control and processing system 400 is respectively connected by data lines with the instantaneous camera 203 of the instantaneous imaging system 200 and the streak camera 305 of the encoding and compression imaging system 300;

[0049] The digital delay pulse generator 401 of the control and processing system 400 is respectively connected by data lines with the instantaneous camera 203 of the instantaneous imaging system 200 and the streak camera 305 of the encoding and compression imaging system 300.

[0050] The present invention works as follows:

[0051] Refer to Figure 1 and Figure 2 : The dynamic scene 101 collected by the objective lens 102 of the dynamic scene acquisition system 100 is further split into two paths by the beam splitter 201 of the instantaneous imaging system 200. One reflected path passes through the first convex lens 202 and is imaged onto the instantaneous camera 203, and instantaneous sampling is extracted through time slicing , obtaining a series of short-exposure, high-quality instantaneous images, providing discrete spatio-temporal information. One transmitted path of the beam splitter 201 passes through the fourth convex lens 301 of the coded compression imaging system 300 to the chrome glass mask 302 with a pseudo-random binary pattern for spatial coding , and the coded scene is relayed through the third convex lens 303 and the second convex lens 304 to the streak camera 305 for shooting, and through time shearing , spatio-temporal integration processing, obtaining the coded compression image. All cameras on the two paths are synchronized by the digital delay pulse generator 401 of the control and processing system 400 to ensure that the same dynamic scene is recorded simultaneously.

[0052] Refer to Figure 1 and Figure 3 : The computer 402 of the control and processing system 400 processes and reconstructs the data collected by the instantaneous camera 203 and the streak camera 305, and uses a dual-modal information fusion image reconstruction algorithm based on a non-trained neural network for reconstruction. This algorithm consists of a neural network with updatable parameters and a physical model of the imaging process. Total variation constraint (TV) is used to constrain the generated image and suppress noise, and the loss function is used as a criterion for updating the neural network parameters.

[0053] Embodiment

[0054] In this embodiment, a high-speed dynamic scene is designed. This embodiment aims to verify the high fidelity of the reconstructed image of the device at high imaging speed by observing the dynamically switching high-speed scene.

[0055] Refer to Figure 1 , the compressed ultrafast imaging device based on instantaneous image reference described in this embodiment includes a dynamic scene acquisition system 100, an instantaneous imaging system 200, a coded compression imaging system 300, and a control and processing system 400. Refer to Figure 4 , the dynamic scene 101 of the dynamic scene acquisition system 100 includes a nanosecond laser, a first reflector, a second reflector, a first beam splitter, a second beam splitter, a first convex lens, a second convex lens, a first mask, and a second mask.

[0056] The nanosecond laser generates a laser beam with a wavelength of 532 nanometers and a pulse width of 10 nanoseconds.

[0057] The function of the first beam splitter is to receive the beam emitted by the nanosecond laser and split it into two beams of the same nature. One beam is transmitted and continues to propagate along the original direction, and the other beam is reflected and propagates perpendicular to the original direction.

[0058] The first mask is a mask that only transmits through a horizontal hole. Its function is to receive and transmit the beam transmitted by the first beam splitter.

[0059] The function of the first mirror is to receive and reflect the beam reflected by the first beam splitter.

[0060] The second mask is a mask that only transmits through a vertical hole. Its function is to receive and transmit the beam reflected by the first mirror.

[0061] The function of the second mirror is to receive and reflect the beam transmitted by the second mask.

[0062] The function of the second beam splitter is to receive and transmit the beam transmitted by the first mask, and receive and reflect the beam reflected by the second mirror. These two beams are combined into the same beam.

[0063] The functions of the first convex lens and the second convex lens are to receive and transmit the beam combined by the second beam splitter.

[0064] The function of the objective lens 102 is to receive the beam transmitted by the first convex lens and the second convex lens, and transmit it to the subsequent imaging device to provide a high-fidelity reconstructed image.

[0065] The path of the beam reflected by the first beam splitter is extended by 2.1 meters compared to the beam transmitted by the first beam splitter. That is, the time difference between the moments when the mask information recorded by the two beams reaches the objective lens 102 is 7 nanoseconds.

[0066] The instantaneous camera 201 of the instantaneous imaging system 200 in this embodiment uses a pair of image-intensified cameras for transient imaging to record two transient frames.

[0067] The function of the digital delay pulse generator 401 of the control and processing system 400 in this embodiment is to provide a synchronous trigger signal. After receiving the signal, the instantaneous imaging system 200 and the coded compressive imaging system 300 start working simultaneously. After obtaining the instantaneous image and the coded compressive image, a reconstruction algorithm is run in the computer 402 to reconstruct the dynamic scene.

[0068] Refer to Figure 5 , (a) is the reconstruction effect of traditional compressive ultrafast imaging, and (b) is the reconstruction effect of this embodiment. It can be seen that compared with other algorithms of traditional compressive ultrafast imaging, this embodiment has clearer edges and fewer artifacts, and the spatio-temporal crosstalk is also reduced in this embodiment.

Claims

1. A compression ultrafast imaging device based on instantaneous image reference, characterized in that It includes: A dynamic scene acquisition system (100) composed of an objective lens (102) and a dynamic scene (101); the objective lens (102) and the dynamic scene (101) are optically connected in sequence; An instantaneous imaging system (200) composed of an instantaneous camera (203), a first convex lens (202) and a beam splitter (201); the instantaneous camera (203), the first convex lens (202) and the reflected path of the beam splitter (201) are optically connected in sequence; An encoded compressive imaging system (300) composed of a streak camera (305), a second convex lens (304), a third convex lens (303), a chromium glass mask (302) and a fourth convex lens (301); the streak camera (305), the second convex lens (304), the third convex lens (303), the chromium glass mask (302) and the fourth convex lens (301) are optically connected in sequence; A control and processing system (400) composed of a digital delay pulse generator (401) and a computer (402); the digital delay pulse generator (401) is connected to the computer (402) by a data line; The objective lens (102) of the dynamic scene acquisition system (100) is optically connected to the beam splitter (201) of the instantaneous imaging system (200); The transmitted path of the beam splitter (201) of the instantaneous imaging system (200) is optically connected to the fourth concave lens (301) of the encoded compressive imaging system (300); The computer (402) of the control and processing system (400) is respectively connected to the instantaneous camera (203) of the instantaneous imaging system (200) and the streak camera (305) of the encoded compressive imaging system (300) by a data line; The digital delay pulse generator (401) of the control and processing system (400) is respectively connected to the instantaneous camera (203) of the instantaneous imaging system (200) and the streak camera (305) of the encoded compressive imaging system (300) by a data line; The digital pulse generator (401) of the control and processing system (400) controls the input trigger signals of the instantaneous camera (203) and the streak camera (305) to ensure that the instantaneous imaging system (200) and the encoded compressive imaging system (300) simultaneously acquire the dynamic scene in the same period; finally, the computer (402) processes and reconstructs the imaging results; The reconstruction adopts a dual-modal information fusion image reconstruction algorithm based on a non-trained neural network; The dynamic scene system (100) performs optical sampling and simultaneously records the dynamic scene through the instantaneous imaging system (200) and the encoded compressive imaging system (300), including encoded compressive imaging and instantaneous imaging; the image reconstruction algorithm restores the high-fidelity dynamic scene by integrating dual-modal information; In optical sampling, and are the measurement images of coded compressive imaging and instantaneous imaging respectively, represents the dynamic scene to be measured, represents the time-shearing operator, represents the spatial encoding operator, represents the instantaneous sampling operator; the image acquisition process is simply expressed by the following formula as ； ； In image reconstruction, the instantaneous image is copied to have the same number as the reconstructed image, denoted as , and is input into the neural network as the initial content, and then the neural network with randomly initialized parameters is used to process the input image; the output three-dimensional spatio-temporal scene obtains the estimated measurement value through the imaging model; subsequently, the estimated measurement value is compared with the experimental result, and the loss with respect to the total variation constraint TV is calculated; through this process, the image reconstruction is transformed into the optimization of the neural network parameters, and its formula is: ； Among them is the optimal parameter of the neural network corresponding to the final reconstructed image, , are the fidelity term weights corresponding to coded compressive imaging and instantaneous imaging respectively, is the TV constraint term weight, represents the L2 norm, represents the TV norm; the loss is calculated for each iteration, and the model parameters are updated using the alternating direction method of multipliers optimizer; after several iterations, the neural network can reconstruct a high-quality three-dimensional spatio-temporal scene .

Images