Super-resolution double-path time-space domain compression imaging method and device

Abstract

The invention discloses a super-resolution double-path time-space domain compression imaging method and device, and belongs to the field of high-speed high-resolution imaging and compression imaging. The system comprises an imaging objective lens, a beam splitter prism, an imaging lens, a high-spatial-resolution camera, a DMD, a low-speed low-spatial-resolution camera and a computer. The implementation method comprises the steps that a high-spatial-resolution imaging light path collects a single-frame high-spatial-resolution image, in a space-time coding compression measurement light path, target light is imaged to the surface of a micromirror array of a DMD through an imaging lens, primary space compression sampling of a target scene is completed at the DMD, and the target scene is subjected to spatial compression measurement; and meanwhile, the DMD controls the micromirror to turn over according to a pre-loaded time-space domain joint coding modulation matrix, so that time-space domain joint modulation of incident target light information is realized. And the low-speed low-spatial-resolution camera performs secondary spatial compression on the modulated multi-frame image, and performs integration on the multi-frame modulated image within single exposure time to obtain a single-frame low-spatial-resolution image. And the obtained high-spatial-resolution image and the space-time compressed image are solved through a reconstruction algorithm to reconstruct the high-spatial-temporal-resolution image.

Description

Technical Field

[0001] This invention relates to a method for high-speed, high-resolution imaging, and particularly to a method and apparatus for high-speed super-resolution reconstruction of dual-channel spatiotemporal compressed images, belonging to the fields of high-speed, high-resolution imaging and compressed imaging. Background Technology

[0002] High-speed, high-resolution imaging is a continuously pursued goal in the development of imaging systems. It aims to acquire high spatial resolution images of moving target scenes at high speed, simultaneously capturing rich spatial details and high-speed motion details. With advancements in materials science and manufacturing processes, both high spatial resolution imaging and high-speed imaging detectors have made rapid progress. However, limited by the bandwidth of the imaging system and the constraints of backend data storage and processing equipment, the temporal and spatial resolutions of the imaging system are often mutually restrictive, thus limiting the development of high-speed, high-resolution imaging detectors. Improving the spatiotemporal resolution of the imaging system simultaneously through hardware upgrades is difficult and costly. While some high-performance cameras can achieve high-speed, high-resolution imaging, these cameras are often expensive. More importantly, the massive amounts of data generated by high-speed, high-resolution imaging make continuous operation difficult for these cameras. Therefore, simultaneously improving the spatiotemporal resolution of the system based on current imaging equipment is of great significance.

[0003] With the continuous development of computer technology, computational imaging has become an effective means to improve the performance of imaging systems and has attracted widespread attention from researchers. Compressed imaging technology is a computational imaging method based on the principle of compressed sensing. Unlike the traditional imaging system's approach of acquiring data first and then compressing it, compressed imaging technology introduces higher-resolution optical modulators into the imaging system to encode and compress the target scene, and then reconstructs the original high-resolution image through a back-end reconstruction algorithm. This method of completing data compression during data acquisition reduces the requirements on detector resolution and readout circuit speed. In compressed imaging systems, the resolution of the high-resolution optical modulator determines the system's maximum resolution; for higher-performance imaging, the performance of the optical modulator will be a limitation. Digital micromirror devices (DMDs) have become typical spatial optical modulators in compressed imaging systems due to their high performance and operability; however, their physical resolution and micromirror flipping speed are mutually constrained, which will pose a challenge in higher-resolution imaging applications.

[0004] Spatiotemporal compression imaging utilizes the spatiotemporal correlation of time-varying images to compress target images simultaneously in both the temporal and spatial domains. Treeaporn V et al., in their paper *Space-time compressive imaging*, discussed the concept and methods of spatiotemporal compression imaging and conducted simulation experiments, but did not establish a practically feasible physical imaging model. Ke Jun et al.'s paper *Temporal Compressed Measurements for Block-wise Compressive Imaging* combines temporal and spatial compression imaging, avoiding the system's temporal resolution degradation caused by the multi-frame measurement requirements in spatial compression imaging; however, this method cannot simultaneously improve both the camera's temporal and spatial resolution. Subsequently, Ke Jun et al. implemented a multi-compression-ratio spatiotemporal compression imaging method in their paper *Multiple CR Spatiotemporal Compressive Imaging System*, but due to its use of single-channel compression measurement, further improvements in reconstructed image quality and spatiotemporal resolution remain limited. Summary of the Invention

[0005] To address the limitations of data bandwidth in high spatiotemporal resolution imaging systems and the limitation of the highest resolution in spatiotemporal compressed sensing imaging by the resolution of optical modulation devices, this invention aims to provide a method and apparatus for super-resolution dual-path spatiotemporal domain compressed imaging. This method enables spatiotemporal domain compressed measurement of a target scene. The introduction of a high spatial resolution imaging optical path provides prior guidance for the reconstruction of super-resolution images with a resolution higher than that of DMD. The acquired spatiotemporal domain encoded compressed image and the high-resolution image acquired by the high-resolution imaging optical path can be used to recover a high-speed, high-resolution image through a recovery algorithm.

[0006] The objective of this invention is achieved through the following technical solution.

[0007] This invention discloses a super-resolution dual-channel spatiotemporal compression imaging method, comprising the following steps:

[0008] Step 1: The target scene passes through the front imaging objective lens and enters the beam splitter prism. One path is reflected by the beam splitter prism and imaged onto a high spatial resolution camera, while the other path passes through the beam splitter prism and enters the spatiotemporal coding compression measurement optical path. This optical path includes a DMD, an imaging lens, and a low-speed, low spatial resolution camera. The target moving scene's acquired images are high-speed, high-resolution image sequences. These high-speed, high-resolution image sequences are divided into image subsequences with the same number of frames in the time dimension. Each image subsequence is represented as... , where R represents the set of real numbers, H and W represent the vertical and horizontal spatial resolutions of the high-resolution image. T represents T consecutive frames of images.

[0009] Step 2: Generate a spatiotemporal joint modulation template to perform spatiotemporal joint modulation on the target light. Considering both the coding efficiency of the coding matrix and hardware feasibility, a random binary matrix is ​​used as the spatiotemporal joint modulation template, denoted as... Where R represents the set of real numbers, and the superscript... This represents the data dimension; in the time dimension, T = K + 1.

[0010] Step 3: First, the high-resolution camera is triggered to begin exposure via a signal generator. The high-resolution camera then acquires the first frame of the target image sequence. The high-resolution camera receives the trigger signal and begins exposure, controlling the exposure time of the high-resolution camera. The high-resolution camera acquires the first frame of the original image sequence, matching the temporal resolution of the target high-resolution image. First frame image Recorded as .

[0011] Step 4: After the high-resolution camera finishes its exposure time, a trigger signal activates the DMD and the low-speed, low-spatial-resolution camera. The DMD flips according to the pre-loaded spatiotemporal joint modulation template generated in Step 2, performing temporal modulation of the incident light. The K frames in the target image sequence, excluding the first frame, are denoted as... ,image When imaging to a DMD, because the spatial resolution of the DMD is smaller than the spatial resolution of the target scene, therefore... Imaging is first performed using a spatial compression sampling method during DMD. The average pooling model is used to represent the compression sampling effect of DMD on the target scene, expressed as:

[0012] (1)

[0013] in express Spatial average pooling compression sampling. in:

[0014] (2)

[0015] This represents the spatial compression sampling factor on the DMD.

[0016] Simultaneously, the DMD controls the flipping of the micromirror array based on the loaded K spatiotemporal joint modulation templates, thereby achieving [the desired effect]. The air conditioning system was set to a specific time, and the modulation result was recorded as follows: The mathematical representation is as follows:

[0017] (3)

[0018] In the formula, k represents the k-th frame of the image sequence or modulation template sequence, i.e. This represents the k-th frame image in a compressed image sequence. Represents the spatiotemporal joint modulation template The k-th frame modulation template in This represents the dot product of matrices, which is the product of corresponding pixels in two matrices.

[0019] Low-speed, low-resolution cameras have lower spatial resolution and frame rate than DMDs. During a single exposure of a low-speed, low-resolution camera, the DMD performs K time-varying modulations on the target scene. The modulation results of the DMD... The image is captured by the imaging lens and projected onto the low-speed, low-resolution detector. The low-speed, low-resolution camera captures the image within a single exposure time. Integrating the data yields a single frame image. Spatially, because the spatial resolution of a low-speed, low-resolution camera is less than that of a DMD, middle Spatial compression is achieved by imaging spatially adjacent pixels onto a single pixel of a low-speed, low-resolution detector. The low-speed, low-resolution detector then acquires spatiotemporal compression measurement images. ,in The mathematical representation of a single pixel in a spatiotemporal compression measurement image is as follows:

[0020] (4)

[0021] In the formula, superscript Indicates the spatial coordinates in the compressed measurement. Indicates will After dividing the space into blocks, the coordinates of each block correspond to the spatial coordinates of the pixels in the compressed measurement values. (Superscript) Represents the coordinates of pixels within a single block of the modulated image, where This represents the spatial coordinates of a pixel within a single block. The time coordinates of a pixel are then... express In the modulated image of the k-th frame, the coordinates of the block with spatial location (p,q) are (i,j), where (i,j) are the coordinates inside the block. Indicates continuous modulation of the image Pixels within the same spatial location block of a frame are summed in both spatiotemporal dimensions. (Relative to...) , The spatiotemporal compression ratio (STCR) is expressed as:

[0022] (5)

[0023] Step 5: Use the backend reconstruction algorithm to reconstruct the single-frame high-resolution image acquired in Step 3. And the spatiotemporal compression measurement values ​​collected in step four. Reconstruct the original high spatiotemporal resolution image. The acquisition process of the image sequence is a continuous image acquisition process; therefore, during reconstruction, not only the first high-resolution measurement image of the current image subsequence can be used, but also... Alternatively, the first frame of the next image subsequence can be used to measure the image. Its reconstruction is represented as:

[0024] (6)

[0025] in This represents the reconstruction algorithm.

[0026] This invention also discloses a super-resolution dual-path spatiotemporal compression imaging device for realizing a high-quality spatiotemporal domain joint compression sensing imaging method. The super-resolution dual-path spatiotemporal domain compression imaging device includes an imaging objective, a beam splitter prism, an imaging lens, a high spatial resolution camera, a DMD, a low-speed, low spatial resolution camera, and a computer. Light from the target first passes through the imaging objective and then enters the beam splitter prism, where it is split into two paths. One path is reflected by the beam splitter prism and collected by the high spatial resolution camera, while the other path passes through the beam splitter prism and is imaged onto the DMD for spatial compression sampling and spatiotemporal domain joint modulation. The imaging fields of view of the high spatial resolution camera detector and the DMD image plane are aligned. The spatial resolution of the DMD is smaller than the spatial resolution of the target image, resulting in spatial compression of the target image on the DMD. The target light signal is modulated according to the loaded spatiotemporal domain joint modulation template. The modulated target beam is then imaged by an imaging lens onto a low-speed, low-resolution camera. Since the spatial and temporal resolutions of the low-speed, low-resolution camera are both smaller than those of the DMD, the low-speed, low-resolution camera performs secondary spatial and temporal compression on the DMD-modulated image sequence to obtain a spatiotemporally compressed image. The obtained high spatial resolution image and spatiotemporally compressed image are uploaded to a computer, and a high spatiotemporal resolution image is reconstructed through a reconstruction algorithm.

[0027] The function of an imaging objective lens is to image the target onto a predetermined image plane.

[0028] The function of the beam splitter is to split the target beam passing through the imaging objective lens into two paths, which reach different image planes. One path enters the high spatial resolution camera for imaging, and the other path enters the spatiotemporal coding compression measurement optical path.

[0029] The purpose of a high spatial resolution camera is to obtain high spatial resolution images.

[0030] The function of the DMD is to load a spatiotemporal joint coding template to modulate the target beam imaged onto the DMD.

[0031] The function of the imaging lens is to image the DMD-modulated light beam onto a low-speed, low-spatial-resolution camera.

[0032] The function of a low spatial resolution camera is to receive modulated light beams and obtain a spatiotemporally compressed image.

[0033] The computer's role is to run a super-resolution dual-path spatiotemporal compression imaging restoration algorithm. It feeds the spatiotemporal joint modulation template, the high spatial resolution image, and the spatiotemporally compressed image into the reconstruction algorithm. The spatiotemporal joint modulation template serves as template prior information, and the high spatial resolution image serves as target prior information. Together, they guide the high-speed, high-resolution reconstruction of the spatiotemporally compressed image. Beneficial effects:

[0034] 1. The present invention discloses a super-resolution dual-channel spatiotemporal domain compression imaging method and apparatus, which utilizes the spatial and temporal correlation of time-varying images and uses a dual-channel structure to acquire high spatial resolution images as prior information and spatiotemporal domain compression measurement values ​​as the measurement values ​​to be reconstructed, thereby achieving super-resolution high-speed imaging with only a small amount of data acquisition, significantly reducing the amount of data acquired, stored and transmitted.

[0035] 2. This invention discloses a super-resolution dual-channel spatiotemporal domain compression imaging method and apparatus. The high-resolution measurement optical path in the dual-channel data acquisition system has a higher spatial resolution than the spatial light modulation device, thus overcoming the limitation of system resolution imposed by the spatial light modulation device. Furthermore, dual-channel information acquisition improves the robustness of the spatiotemporal domain compression imaging system to various application scenarios and enhances the image reconstruction quality. High spatiotemporal resolution images can be acquired using a beam splitter prism, two cameras, a DMD, an imaging lens, and control equipment.

[0036] 3. This invention discloses a super-resolution dual-path spatiotemporal domain compression imaging method and apparatus. In the spatiotemporal coding compression measurement optical path, the target light is imaged onto the micromirror array surface of the DMD via an imaging lens. Since the spatial resolution of the DMD is lower than that of the target, a spatial compression sampling of the target scene is completed at the DMD. Simultaneously, the DMD controls the micromirror flipping according to a pre-loaded spatiotemporal domain joint coding modulation matrix, realizing spatiotemporal domain joint modulation of the incident target light information. The spatiotemporally domain joint modulated target light information continues to be imaged onto a low-speed, low-spatial-resolution camera via a rear imaging lens. The low-speed, low-spatial-resolution camera performs secondary spatial compression on the modulated multi-frame images and integrates the multi-frame modulated images within a single exposure time to obtain a single-frame low-spatial-resolution image, thereby realizing spatiotemporal domain compression measurement of the target scene, reducing the amount of data acquisition, and thus reducing the data bandwidth requirements of the system for high-speed, high-resolution imaging. Attached Figure Description

[0037] Figure 1 is a schematic diagram of the principle of the super-resolution dual-channel spatiotemporal compression imaging method;

[0038] Figure 2 is a structural diagram of the system.

[0039] Figure 3 is a schematic diagram of the image modulation and acquisition process of super-resolution dual-channel spatiotemporal joint compressed sensing imaging.

[0040] Figure 4 is a schematic diagram of the control signals in the embodiment;

[0041] Figure 5 shows a high-speed, high-resolution image reconstructed from simulated data in the embodiment;

[0042] Among them, 1-moving target or scene, 2-imaging objective, 3-beam splitter prism, 4-digital micromirror array (DMD), 5-imaging lens, 6-low-speed, low-resolution camera, 7-high-resolution camera, and 8-computer. Detailed Implementation

[0043] To better illustrate the purpose and advantages of the present invention, the invention will be further described below in conjunction with the accompanying drawings and examples.

[0044] like Figure 1 As shown, the original high-resolution image subsequence Nine consecutive frames of images with a spatial resolution of 1024×1024, a frame rate of 120fps, and a pixel depth of 8 bits are generated. The first frame is denoted as P1, and frames 2-9 are denoted as P2. P3 is the first frame of the next image subsequence. The target scene passes through the front imaging objective lens into the beam splitter, and is reflected by the beam splitter to the high-resolution camera. The spatial resolution camera has an exposure time of (1 / 120)s, acquiring the first frame of the current image subsequence at the original resolution of 1024×1024. The image sequence passes through a beam splitter into the spatiotemporal encoding compression measurement optical path. Imaging to the DMD completes one spatial compression and achieves joint spatiotemporal modulation; the modulation result is denoted as... Spatial compression sampling factor of DMD Then the imaging lens images the modulated image sequence onto a low-speed, low-resolution camera, increasing the spatial compression factor of the detector. With a temporal compression factor K=8, a single-frame spatiotemporal compressed measurement image with a spatial resolution of 256×256 is finally obtained. The spatiotemporal compression ratio is:

[0045] (7)

[0046] Since the image sequence acquisition process is continuous, the reconstruction algorithm can utilize the high-resolution measurement value of the first frame of the next image sequence, which is shown in the figure. and Simultaneously, as prior information, along with spatiotemporal compressed measurements, the original high-resolution image sequence is reconstructed using a back-end reconstruction algorithm. .

[0047] Step 1: The target scene passes through the front imaging objective lens and enters the beam splitter prism. One path is reflected by the beam splitter prism and imaged onto the high-resolution camera, while the other path passes through the beam splitter prism and enters the spatiotemporal coding compression measurement optical path. This optical path includes the DMD, imaging lens, and low-speed, low-resolution camera. The target moving scene's acquired images are high-speed, high-resolution image sequences. These high-speed, high-resolution image sequences are divided into image subsequences with the same number of frames in the time dimension. Each image subsequence is represented as... Where R represents the set of real numbers, H and W represent the vertical and horizontal spatial resolutions of the high-resolution image, and T represents T consecutive frames of images. In this embodiment, H=W=1024 and T=9.

[0048] Step 2: Generate a spatiotemporal joint modulation template to perform spatiotemporal joint modulation on the target light. Considering both the coding efficiency of the coding matrix and hardware feasibility, a random binary matrix is ​​used as the spatiotemporal joint modulation template, denoted as... Where R represents the set of real numbers, and the superscript... This represents the data dimension; in the time dimension, T = K + 1. In this embodiment, = =512, K=8.

[0049] Step 3: First, the high-resolution camera is triggered to begin exposure via a signal generator. The high-resolution camera then acquires the first frame of the image sequence. The high-resolution camera receives the trigger signal and begins exposure, controlling the exposure time of the high-resolution camera. The high-resolution camera acquires the first frame of the original image sequence, matching the temporal resolution of the target high-resolution image. First frame image Recorded as In this embodiment =(1 / 120)s, H=W=1024.

[0050] Step 4: After the high-resolution camera finishes its exposure time, a trigger signal activates the DMD. The DMD then flips according to the pre-loaded spatiotemporal joint modulation template generated in Step 2, performing temporal modulation on the incident light. The K=8 frames in the target image sequence, excluding the first frame, are denoted as... Perform spatiotemporal coding, image When imaging to a DMD, because the spatial resolution of the DMD is smaller than the spatial resolution of the target scene, therefore... Imaging is first performed using a spatial compression sampling method during DMD. The average pooling model is used to represent the compression sampling effect of DMD on the target scene, expressed as:

[0051] (8)

[0052] in express Spatial average pooling compressed sampling. In this embodiment... =2, In this embodiment:

[0053] (9)

[0054] Simultaneously, the DMD controls the flipping of the micromirror array based on the loaded K=8 spatiotemporal joint modulation templates, thereby achieving [the desired effect]. The air conditioning system was set to a specific time, and the modulation result was recorded as follows: The mathematical representation is as follows:

[0055] (10)

[0056] In the formula, k represents the k-th frame of the image sequence or modulation template sequence, i.e. This represents the k-th frame image in a compressed image sequence. Represents the spatiotemporal joint modulation template The k-th frame modulation template in This represents the dot product of matrices, which is the product of corresponding pixels in two matrices.

[0057] Low-speed, low-resolution cameras have lower spatial resolution and frame rate than DMDs. In a single exposure of a low-speed, low-resolution camera, the DMD performs K time-varying modulations on the target scene. The modulation results of the DMD... The image is captured by the imaging lens and projected onto the low-speed, low-resolution detector. The low-speed, low-resolution camera captures the image within a single exposure time. Integrating the data yields a single frame image. Spatially, because the spatial resolution of a low-speed, low-resolution camera is less than that of a DMD, [the following applies]. Divide the data into blocks, with a block size of [size missing]. Spatial compression is achieved by imaging individual blocks onto a single pixel of a low-speed, low-resolution detector. The low-speed, low-resolution detector acquires spatiotemporally compressed measurement images. ,in In this embodiment =2, The mathematical representation of a single pixel in a spatiotemporal compression measurement image is as follows:

[0058] (11)

[0059] In the formula, superscript Indicates the spatial coordinates in the compressed measurement. Indicates will After dividing the space into blocks, the coordinates of each block correspond to the spatial coordinates of the pixels in the compressed measurement values. (Superscript) Represents the coordinates of pixels within a single block of the modulated image, where This represents the spatial coordinates of a pixel within a single block. The time coordinates of a pixel are then... express In the modulated image of the k-th frame, the coordinates of the block with spatial location (p,q) are (i,j), where (i,j) are the coordinates inside the block. Indicates continuous modulation of the image Pixels within the same spatial location block of a frame are summed in both spatiotemporal dimensions. (Relative to...) In this embodiment The spatiotemporal compression ratio (STCR) is expressed as:

[0060] = =128 (12)

[0061] Step 5: Use the backend reconstruction algorithm to reconstruct the single-frame high-resolution image acquired in Step 3. And the spatiotemporal compression measurement values ​​collected in step four. Reconstruct the original high spatiotemporal resolution image. The acquisition process of the image sequence is a continuous image acquisition process; therefore, during reconstruction, not only the first high-resolution measurement image of the current image subsequence can be used, but also... Alternatively, the first frame of the next image subsequence can be used to measure the image. The embodiment uses a deep learning-based spatiotemporal domain reconstruction algorithm, utilizing high-resolution guiding frames. and spatiotemporal compression measurement frames Reconstructing the original high-speed, high-resolution image using spatiotemporal joint modulation template M The image restoration algorithm employs deep learning, with a network model based on the STFormer model. A preprocessing module is added for measurements consisting of high-resolution guided images and spatiotemporally compressed measurements. This ultimately achieves high-speed, super-resolution reconstruction based on high-resolution guided images and spatiotemporally compressed measurements. The reconstructed image is represented as follows:

[0062] (13)

[0063] like Figure 5As shown, in this embodiment, eight high-resolution images with a spatial resolution of 1024×1024 can be reconstructed from a single 256×256 measurement image. Therefore, this embodiment uses a dual-path spatiotemporal compression imaging method to achieve a spatial resolution of 16 for the compressed image. A 100-fold improvement, with an 8-fold improvement in temporal resolution. The restoration effect is as follows... Figure 5 As shown, the spatiotemporal domain coded compression measurements and the high-speed, high-resolution reconstruction results based on high spatial resolution guided images and spatiotemporal domain coded compression measurements are presented respectively. To quantitatively analyze the reconstruction effect, the peak signal-to-noise ratio (PSNR) of the reconstructed images was calculated. PSNR is defined as follows:

[0064] (14)

[0065] (15)

[0066] Where MSE represents the mean squared error. This represents the k-th frame image in the original high-resolution image sequence. Represents the reconstructed image sequence The reconstructed image of the k-th frame in the image, where MAX represents the maximum pixel value of the image.

[0067] Apparatus for implementing the above method, such as Figure 2 As shown, the system includes an imaging objective 2, a beam splitter 3, a DMD 4, an imaging lens 5, a low-speed, low-resolution camera 6, a high spatial resolution camera 7, and a computer 8. Light from the moving target 1 first passes through the imaging objective 2, and after reaching the beam splitter 3, it is split into two paths. One path is reflected by the beam splitter 3 into the high spatial resolution imaging optical path, and the other path passes through the beam splitter 3 into the spatiotemporal coding compression measurement optical path. Under the action of a control signal, the high-resolution camera 7 acquires the first frame image of the target sequence, obtaining a high spatial resolution image. The beam entering the spatiotemporal coding compression measurement optical path is imaged onto the DMD 4. The DMD performs spatiotemporal joint modulation of the target beam according to a preset loading coding template. The modulated target beam then passes through the imaging lens 5 and is imaged onto the low-speed, low-resolution camera 6. Since the spatiotemporal resolution of the low-speed, low-resolution camera is smaller than that of the DMD, the low-resolution camera 6 obtains a two-dimensional spatiotemporally compressed image. The obtained high-resolution image and compressed image are uploaded to the computer 8, and a high-speed, high-resolution original image is reconstructed through a reconstruction algorithm.

[0068] The function of imaging objective 2 is to image the target.

[0069] The function of beam splitter 3 is to split the target beam into two paths: one path enters the high spatial resolution imaging optical path, and the other path enters the spatiotemporal coding compression measurement optical path.

[0070] The function of DMD 4 is to store and load the designed encoding template, and to perform time-limited control on the target beam imaged onto the DMD.

[0071] The function of imaging lens 5 is to image the beam reflected by the DMD onto a low-speed, low-resolution camera.

[0072] The function of the low-speed, low-resolution camera 6 is to receive the modulated light beam and obtain a compressed image.

[0073] The high spatial resolution camera 7 is designed to obtain images with high spatial resolution.

[0074] The role of computer 8 is to run the spatiotemporal compression imaging restoration algorithm. It feeds the spatiotemporal joint modulation template, the high spatial resolution image, and the spatiotemporal compressed image into the reconstruction algorithm. The spatiotemporal joint modulation template serves as template prior information, and the high spatial resolution image serves as target prior information. The two together guide the high-speed, high-resolution reconstruction of the spatiotemporal compressed image.

[0075] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A super-resolution dual-channel spatiotemporal compression imaging method, characterized in that: Includes the following steps: Step 1: The target scene enters the beam splitter through the front imaging objective lens. One path is reflected by the beam splitter and imaged to the high spatial resolution camera; the other path passes through the beam splitter and enters the spatiotemporal coding compression measurement optical path, which includes a DMD, an imaging lens, and a low-speed, low spatial resolution camera. The target acquisition image of the target moving scene is a high-speed, high-resolution image sequence. The target high-speed, high-resolution image sequence is divided into image subsequences with the same number of frames in the time dimension. Step 2: Generate a spatiotemporal joint modulation template to perform spatiotemporal joint modulation on the target light; Step 3: First, the high-resolution camera is triggered to start exposure via the signal generator, and the high-resolution camera acquires the first frame of the original image sequence. Step 4: After the high-resolution camera finishes its exposure time, a trigger signal activates both the DMD and the low-speed, low-resolution camera. The DMD then flips the images according to the pre-loaded spatiotemporal joint modulation template generated in Step 2. The DMD then processes the K frames in the image sequence, excluding the first frame. Spatiotemporal coding is performed to obtain the modulation result; a low-speed, low-resolution camera performs spatiotemporal compression on the modulation result to obtain a spatiotemporally compressed measurement image.

2. The method as described in claim 1, characterized in that: Each image subsequence in step one is represented as follows: , where R represents the set of real numbers, H and W represent the vertical and horizontal spatial resolutions of the high-resolution image; T represents T consecutive frames of images.

3. The method as described in claim 1, characterized in that: Step two is implemented by using a random binary matrix as the spatiotemporal joint modulation template, denoted as... Where R represents the set of real numbers, and the superscript... This represents the data dimension; in the time dimension, T = K + 1.

4. The method as described in claim 1, characterized in that: Step three is implemented by having the high-resolution camera first receive a trigger signal to begin exposure, and then controlling the exposure time of the high-resolution camera. The high-resolution camera acquires the first frame of the original image sequence, matching the temporal resolution of the target high-resolution image. First frame image Recorded as .

5. The method as described in claim 1, characterized in that: The method for obtaining the modulation result in step four is as follows: image When imaging to the DMD, since the spatial resolution of the DMD is smaller than that of the target scene, spatial compression sampling occurs during imaging in the DMD. The average pooling model is used to represent the compression sampling effect of the DMD on the target scene, expressed as: （1） in express Spatial average pooling compressed sampling; in: （2） The spatial compression sampling factor on the DMD; Because the frame rate of a low-speed, low-resolution camera is less than that of a DMD, the frequency of the DMD trigger signal is K times the frequency of the low-speed, low-resolution camera. That is, within a single exposure time of the low-speed, low-resolution camera, the DMD loads K spatiotemporal joint modulation templates. The DMD loads these K templates, receives the trigger signal, and controls the micromirror array's movement according to the templates, thus achieving [the desired effect]. The air conditioning system was set to a specific time, and the modulation result was recorded as follows: It is expressed as follows: （3） In the formula, k represents the k-th frame of the image sequence or modulation template sequence, i.e. This represents the k-th frame image in a compressed image sequence. Represents the spatiotemporal joint modulation template The k-th frame modulation template in This represents the dot product of matrices, which is the product of corresponding pixels in two matrices.

6. The method as described in claim 1, characterized in that: The method for obtaining the spatiotemporal compression measurement image in step four is as follows: The imaging lens projects the DMD-modulated image onto a low-speed, low-resolution camera. Since the spatiotemporal resolution of the low-speed, low-resolution camera is smaller than that of the DMD, during the imaging process, the light reflected by multiple micromirrors of the DMD is projected onto a single pixel of the low-speed, low-resolution camera through the imaging lens, thus achieving spatial compression sampling. Within a single exposure time of the camera, multiple modulated images are integrated to achieve temporal compression measurement; the spatiotemporal compressed measurement image obtained within a single exposure time of the camera is denoted as... , in: （4） Indicates the adjacent DMDs during the imaging process. The light reflected by the micromirror array is imaged onto a single pixel of the camera; Spatiotemporal compression measurement images The mathematical representation of a single pixel is as follows: （5） In the formula, superscript Indicates the spatial coordinates in the compressed measurement; subscript This indicates that the coordinates of each block after the modulated image is divided into blocks correspond to the spatial coordinates of pixels in the compressed measurement; superscript Represents the coordinates of pixels within a single block of the modulated image, where This represents the spatial coordinates of a pixel within a single block. The time coordinates of a pixel are then... express In the modulated image of the k-th frame, the value of the pixel with coordinates (i,j) in the block with spatial location (p,q), where (i,j) is the coordinate inside the spatial block; Indicates continuous modulation of the image Pixels within the same spatial location block of a frame are summed in both spatiotemporal dimensions; relative to... , The spatiotemporal compression ratio (STCR) is expressed as: （6） Step 5: Use the backend reconstruction algorithm to reconstruct the single-frame high-resolution image acquired in Step 3. And the spatiotemporal compression measurement values ​​collected in step four. Reconstructing the original high spatiotemporal resolution image; the image sequence acquisition process is a continuous image acquisition process, therefore, during reconstruction, not only can the first high-resolution measurement image of the current image subsequence be used, but also... It can also measure images using the first frame of the next image subsequence. Its reconstruction is represented as: （7） in This represents the reconstruction algorithm function.

7. An apparatus for implementing the method as described in claims 1 to 6, characterized in that: It includes an imaging objective, a beam splitter, an imaging lens, a high spatial resolution camera, a DMD, a low-speed low spatial resolution camera, and a computer. Light from the target first passes through the imaging objective and then enters the beam splitter, where it is split into two paths. One path is reflected by the beam splitter and collected by the high spatial resolution camera, while the other path is imaged through the beam splitter onto the DMD for spatial compression sampling and spatiotemporal joint modulation. The imaging fields of the high spatial resolution camera detector and the DMD image plane are aligned. The DMD performs spatiotemporal modulation on the target light signal according to the loaded spatiotemporal joint modulation template. The modulated target beam is then imaged by the imaging lens onto a low-speed, low-spatial-resolution camera. The low-speed, low-resolution camera performs secondary spatial and temporal compression on the DMD-modulated image sequence to obtain a spatiotemporally compressed image. The obtained high spatial resolution images and spatiotemporally compressed images are uploaded to a computer, and high spatiotemporal resolution images are reconstructed by solving the reconstruction algorithm.

8. The apparatus as described in claim 7, characterized in that: The function of an imaging objective lens is to image the target onto a predetermined image plane; The function of the beam splitter is to split the target beam passing through the imaging objective into two paths, which reach different image planes respectively. One path enters the high spatial resolution camera for imaging, and the other path enters the spatiotemporal coding compression measurement optical path. The purpose of a high spatial resolution camera is to obtain high spatial resolution images; The function of the DMD is to load a spatiotemporal joint coding template and perform spatiotemporal control on the target beam imaged onto the DMD. The function of the imaging lens is to image the DMD-modulated light beam onto a low-speed, low-resolution camera. The function of a low spatial resolution camera is to receive a temporally compressed light beam and obtain a spatiotemporally compressed image. The computer is used to run a super-resolution dual-channel spatiotemporal domain compression imaging restoration algorithm. The spatiotemporal domain joint modulation template, high spatial resolution image, and spatiotemporal compressed image are fed into the reconstruction algorithm. The spatiotemporal domain joint modulation template serves as template prior information, and the high spatial resolution image serves as target prior information. The two together guide the high-speed, high-resolution reconstruction of the spatiotemporal compressed image.

Images