A light field imaging device and method with adjustable angular resolution and imaging time.

By combining a front imaging lens group, a Fourier transform lens group, a beam splitter, a spatial light modulator, and an image sensor, a light field imaging device is used to solve the problems of decreased image spatial resolution and excessively long imaging time in light field imaging using single-pixel imaging technology, thus achieving a light field imaging effect with adjustable angular resolution and imaging time.

CN116708744BActive Publication Date: 2026-11-17JINAN UNIVERSITY
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202310448570.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-04-24
Publication Date
2026-11-17
Estimated Expiration
2043-04-24

AI Technical Summary

Technical Problem

In existing light field imaging technologies, the microlens array-based method leads to a decrease in image spatial resolution. The spatial light modulator modulates the target object's image plane, and the image sensor detects the light field signal on the Fourier spectrum, resulting in excessively long imaging times, which cannot meet the imaging requirements of dynamic objects.

Method used

The device employs a combination of a front imaging lens group, a Fourier transform lens group, a beam splitter, a spatial light modulator, and an image sensor. Combined with single-pixel imaging technology, the size of the macro-modulation unit of the spatial light modulator is controlled by a computer to achieve adjustable angular resolution and imaging time. The image sensor directly acquires the two-dimensional spatial information of the target object, and the spatial light modulator modulates the image on the Fourier spectrum.

Benefits of technology

It achieves image spatial resolution up to the full sampling resolution of the image sensor, with adjustable angular resolution and imaging time, adapting to the imaging needs of dynamic objects, and shortening the imaging time without reducing spatial resolution.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116708744B_ABST
    Figure CN116708744B_ABST
Patent Text Reader

Abstract

This invention discloses a light field imaging device and method with adjustable angular resolution and imaging time. A front imaging lens group images the target object onto the vicinity of the front focal plane of a Fourier transform lens group. The light then passes through the Fourier transform lens group and a beam splitter, forming a Fourier spectrum image on the rear focal plane of the Fourier transform lens group. The Fourier spectrum image is modulated by a spatial light modulator, and the modulated light is reflected back to the beam splitter. After passing through the beam splitter, the light passes through the rear imaging lens group, imaging the target object onto the photosensitive surface of an image sensor. The image of the target object is then acquired by the image sensor. Using the acquired signal, the Fourier spectrum image is reconstructed using a single-pixel imaging method to obtain the light angle information. A viewing angle image is then reconstructed, and digital refocusing is performed using the viewing angle image to acquire scene images focused at different depths. The angular resolution and imaging time of the light can be adjusted by computer-controlled macro-modulation unit size of the spatial light modulator.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of optical imaging technology, and more specifically to a light field imaging device and method with adjustable angular resolution and imaging time. Background Technology

[0002] In recent years, with the rapid development of computer technology, digital optoelectronic detection technology, light field manipulation technology, and information processing technology, a new imaging mode—computational imaging—has emerged. Computational imaging technology has overcome many of the insurmountable limitations of traditional direct imaging. Traditional imaging using two-dimensional image sensors cannot directly acquire three-dimensional spatial images of the target object. Light field imaging is a typical computational imaging technique for acquiring three-dimensional spatial information of a target object.

[0003] Currently widely used light field imaging systems primarily acquire signals by inserting a microlens array in front of an image sensor. The function of the microlens array is to multiplex the two-dimensional spatial information of the image and the two-dimensional angular information of the light rays within a single two-dimensional image sensor space. Its advantage is that only a single acquisition is required, which is beneficial for light field imaging of dynamic objects. However, due to the bandwidth product limitation of a single two-dimensional image sensor, this microlens array-based light field signal spatial multiplexing acquisition method suffers from the problem of mutual encroachment between the detection space of two-dimensional spatial information and two-dimensional angular information. This leads to a mutual constraint between the spatial resolution of the image and the angular resolution of the light rays, resulting in a significant decrease in the spatial resolution of the image.

[0004] To address the severe degradation of image spatial resolution caused by microlens array-based methods, Zhong Jingang et al. proposed a light field imaging method that utilizes a spatial light modulator to modulate the image plane of the target object, and then uses an image sensor to acquire the light field signal on the Fourier spectrum [Chinese Invention Patent: Zhong Jingang, Yao Manhong, Huang Zhuobin, Peng Junzheng, A Light Field Camera and Imaging Method, Patent No.: 202110020754.3; and Paper: Yuhang Hu, Manhong Yao, Zhuobin Huang, Junzheng Peng, Zibang Zhang, Jingang Zhong, “Full-Resolution Light-Field Camera via Fourier Dual Photography,” Photonics, 9(8):559, 2022.]. This method does not require a microlens array. Based on the principle of dual photography, the spatial light modulator is equivalent to a virtual image sensor that acquires the two-dimensional spatial information of the image, while the actual image sensor acquires the two-dimensional angular information of the light rays. This method is equivalent to using two image sensors to acquire four-dimensional light field information. There is no issue of mutual encroachment between the detection space of two-dimensional angular information and two-dimensional spatial information, and therefore no mutual constraint between the spatial resolution of the image and the angular resolution of the light rays. The angular resolution of this method has the full sampling resolution of a two-dimensional image sensor. However, because it uses a spatial light modulator to modulate the image plane of the target object, the spatial resolution of the reconstructed image is determined by the spatial modulation resolution (or the size of the modulation unit) of the spatial light modulator. If a single-pixel imaging technique image reconstruction algorithm is used [see paper: Tian'ao Lu, Ziheng Qiu, Zibang Zhang, Jingang Zhong, “Comprehensive comparison of single-pixel imaging methods,” Optics and Lasers in Engineering, Vol. 134, 106301, 2020.] to reconstruct an image of the target object with the full sampling resolution of the spatial light modulator, the number of modulations by the spatial light modulator and acquisitions by the image sensor is at least equal to the number of modulation units of the spatial light modulator. For example, assuming the spatial light modulator has 512×512 modulation units, to obtain a reconstructed image with a resolution of 512×512, at least 512×512 = 262144 modulation and acquisition operations are required. Therefore, acquiring a high-resolution image requires a large number of modulation and acquisition operations, resulting in excessively long imaging times, which is detrimental to imaging dynamic objects.To significantly reduce the number of modulation and acquisition operations and shorten imaging time, it is necessary to drastically reduce the number of modulation units in the spatial light modulator, thereby significantly reducing the spatial modulation resolution of the spatial light modulator. This, in turn, leads to a significant decrease in the spatial resolution of the reconstructed image, and low spatial resolution images are not conducive to the widespread application of the technology. Summary of the Invention

[0005] This invention aims to overcome the shortcomings of the aforementioned microlens array-based light field imaging methods, such as decreased image spatial resolution, and the long imaging time of existing methods involving spatial light modulators modulating the image plane of the target object and image sensors detecting light field signals on the Fourier spectrum. It provides a light field imaging device and method with adjustable angular resolution and imaging time. This invention proposes a novel spatial light modulation and image sensor signal acquisition method, along with a new viewpoint map reconstruction algorithm. This ensures that the lateral spatial resolution of the refocused target object image has the full sampling resolution of the image sensor. The angular resolution of the light rays and the imaging time can be adjusted by computer-controlled macro-modulation unit size of the spatial light modulator. The smaller the macro-modulation unit, the shorter the imaging time; the shortest imaging time can be adjusted to the imaging time of a single acquisition by the image sensor.

[0006] The technical solution adopted in this invention includes:

[0007] In a first aspect, the present invention provides a light field imaging device with adjustable angular resolution and imaging time. The light field imaging device includes: a front imaging lens group 1, a Fourier transform lens group 2, a beam splitter 3, a spatial light modulator 4, a rear imaging lens group 5, an image sensor 6, and a computer 7. The image plane of the front imaging lens group 1 that images the target object coincides with the front focal plane of the Fourier transform lens group 2; the rear focal plane of the Fourier transform lens group 2 coincides with the modulation plane of the spatial light modulator 4; the beam splitter 3 is disposed between the Fourier transform lens group 2 and the spatial light modulator 4; the front focal plane of the rear imaging lens group 5 coincides with the rear focal plane of the Fourier transform lens group 2; the photosensitive surface of the image sensor 6 coincides with the rear focal plane of the rear imaging lens 5; and the computer 7 is connected to both the spatial light modulator 4 and the image sensor 6.

[0008] The light emitted from the target object first passes through the front imaging lens group 1, which images the target object onto the vicinity of the front focal plane of the Fourier transform lens group 2. Then, the light passes through the Fourier transform lens group 2 and the beam splitter 3 in sequence, forming a Fourier spectrum image on the rear focal plane of the Fourier transform lens group 2. The Fourier spectrum image is modulated by the spatial light modulator 4, and the modulated light is reflected back to the beam splitter 3. After passing through the beam splitter 3, the light passes through the rear imaging lens group 5, which images the target object onto the photosensitive surface of the image sensor 6. The image of the target object is then acquired by the image sensor 6.

[0009] The computer 7 is used to control the modulation of the Fourier spectrum image by the spatial light modulator 4 and the synchronous acquisition of the target object image by the image sensor 6. It is also used to generate the modulation pattern of the spatial light modulator 4, analyze and process the image data, reconstruct the refocused image, display the imaging results, and store the data.

[0010] Furthermore, the front imaging lens group 1 includes a focusing device, which images target objects at different distances onto the vicinity of the front focal plane of the Fourier transform lens group 2 through focusing. By changing the distance between the front imaging lens group 1 and the Fourier transform lens group 2, or by changing the focal length of the front imaging lens group 1, focusing on target objects at different distances can be achieved, thereby enabling light field imaging of target objects at different distances.

[0011] Furthermore, the front imaging lens group 1 also includes a microscope objective and a tube mirror, which image the tiny target object onto the vicinity of the front focal plane of the Fourier transform lens group 2. The tiny target object is placed near the front focal plane of the microscope objective, and magnified and imaged by the front imaging lens group composed of the microscope objective and the tube mirror, which is beneficial for achieving light field imaging of the tiny target object.

[0012] Secondly, the present invention provides a light field imaging method based on the above-mentioned light field imaging device with adjustable angular resolution and imaging time, the steps of which are as follows:

[0013] S1. The Fourier spectrum image is modulated: The target object is imaged onto the front focal plane of the Fourier transform lens group 2 via the front imaging lens group 1, and the modulation surface of the spatial light modulator 4 is set on the rear focal plane of the Fourier transform lens group 2, wherein the number of modulation pixels of the spatial light modulator 4 is M. s ×N s M is formed through computer 7 control. P ×N p One macro modulation unit; P1(m) is a K-amplitude spatial light modulation pattern pre-generated by computer 7. p ,n p P2(m) p ,n p ), ..., P k (m p ,n p ), ..., P K (m p ,n p ), which are sequentially loaded onto the spatial light modulator 4 to modulate the Fourier spectrum image sequentially, where P k (m p ,n p Let P be the k-th spatial light modulation pattern. k (m p ,n pThe coordinates of the macro-modulation unit corresponding to spatial light modulator 4 are (m p ,n p The number of modulation pixels in spatial light modulator 4 is M. s ×N s M P ≤M s N p ≤N s , 1≤k≤K, 1≤m p ≤M P 1≤n p ≤N p , k, K, m p n p M p N p M s N s It is a positive integer;

[0014] S2. Target Object Image Acquisition: The Fourier spectrum image of the target object's light field is modulated by the spatial light modulator 4, and then imaged onto the photosensitive surface of the image sensor 6 by the rear imaging lens group 5. The acquisition of the image sensor 6 and the modulation of the spatial light modulator 4 are synchronized. The image sensor acquires K images of the target object modulated by the spatial light modulator 4: I1(m c ,n c ), I2(m c ,n c ), ..., I k (m c ,n c ), ..., I K (m c ,n c ), where: I k (m c ,n c Let I be the k-th image of the target object. k (m c ,n c The pixel coordinates corresponding to the photosensitive surface of image sensor 6 are (m c ,n c ), 1≤m c ≤M c 1≤n c ≤N c The photosensitive surface of image sensor 6 is made of M c ×N c Composed of pixels, m c n c M c N c positive integer

[0015] S3. Fourier Spectrum Image Reconstruction: Using the photosensitive pixels of image sensor 6 as single-pixel detectors, and for each of the K grayscale values ​​detected by the single-pixel detector, an image reconstruction algorithm based on single-pixel imaging technology is used to reconstruct a Fourier spectrum image modulated by spatial light modulator 4, resulting in a total of M reconstructed images. c ×N c Fourier spectrum image:

[0016] S4. Viewpoint Image Reconstruction: Reconstruct the M image from the above-mentioned M... c ×N c M Fourier spectrum images were arranged according to the 6-pixel coordinates of the image sensor. Gray values ​​of the same pixel coordinates were extracted from each image, and then rearranged according to the order of the Fourier spectrum images to form a single viewpoint image, resulting in a total of M images. p ×N p Aperture view image:

[0017] S5. Target object image refocusing and reconstruction: using the M obtained above... p ×N p Take one of the four viewpoint images as a reference and translate all the other viewpoint images. The translation amount is determined by the relative position between the Fourier spectra of each viewpoint image and the Fourier spectra of the reference viewpoint image, as well as the refocusing distance d. Then, superimpose all the translated viewpoint images to obtain a single image of the target object with a refocusing distance of d. By setting different refocusing distances, different images of the target object can be obtained.

[0018] Further, in step S1, the computer 7 will connect the adjacent N in the spatial light modulator 4 ms Each modulation pixel is set as a macro modulation unit to generate M P ×N p There are N macro modulation units, and the size of each macro modulation unit is N. ms The sum of the modulation pixels, and the N of different macro-modulation units. ms Same or different; where N ms The value is a positive integer. The size of the macro-modulation unit determines the angular resolution of the reconstructed optical field. The smaller the macro-modulation unit, the smaller the angular sampling interval, and the higher the angular resolution, but the longer the imaging time. By digitally changing the size of the macro-modulation unit, the angular resolution and imaging time can be adjusted.

[0019] Further, in step S2, the image sensor 6 is a color image sensor, which acquires K color images of the target object modulated by the spatial light modulator. Each color image is then decomposed into three monochrome images: red (R), green (G), and blue (B). The monochrome images are then subjected to Fourier spectrum image reconstruction, viewpoint image reconstruction, and target object image refocusing reconstruction according to steps S3, S4, and S5, respectively. Finally, the three images (R, G, and B) obtained at the same refocusing distance are combined into a single color refocusing image. By utilizing the encoding rules of the three colors R, G, and B in the color image sensor, each color image is decomposed into three monochrome images (R, G, and B). Color light field imaging is beneficial for distinguishing objects of different colors within the target object.

[0020] Furthermore, the K-frame spatial light modulation pattern pre-generated by the computer is a Fourier transform basis pattern. Using an image reconstruction algorithm based on Fourier single-pixel imaging technology, the Fourier spectrum image of the target object's light field is reconstructed. The Fourier transform basis pattern is a sinusoidal distribution pattern, which is advantageous for modulation using an amplitude-type liquid crystal spatial light modulator.

[0021] Furthermore, the K spatial light modulation patterns pre-generated by computer 7 are Hadamard transform basis patterns. Using the image reconstruction algorithm of Hadamard single-pixel imaging technology, the Fourier spectrum image of the light field of the target object is reconstructed. The Hadamard transform basis pattern is a binarized pattern. Using the binarized spatial light modulator 4 to modulate the Hadamard transform basis pattern can achieve higher modulation speed, which is beneficial to reducing imaging time.

[0022] Furthermore, the K spatial light modulation patterns pre-generated by the computer are random patterns. Using an image reconstruction algorithm based on computational ghost imaging single-pixel imaging technology, the Fourier spectrum image of the target object's light field is reconstructed. This image reconstruction algorithm based on computational ghost imaging single-pixel imaging technology has lower computational complexity and faster image reconstruction speed.

[0023] Furthermore, in step S3, the N adjacent pixels in the image sensor 6 are... mc Each photosensitive pixel is set as a macro pixel, and the grayscale value of each macro pixel is N. mc The sum of the gray values ​​of each photosensitive pixel, and the N values ​​of different macropixels. mc Same or different; each macropixel acts as a single-pixel detector. Using the K grayscale values ​​detected by each single-pixel detector, and based on the image reconstruction algorithm of single-pixel imaging technology, each single-pixel detector reconstructs a Fourier spectrum image, where N... mc It is a positive integer. Since the number of macro pixels is less than the total number of pixels in the image sensor 6, it will speed up the image reconstruction speed of light field imaging.

[0024] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0025] This invention utilizes a spatial light modulator 4 to modulate the Fourier spectrum, combined with single-pixel imaging technology. The spatial light modulator 4 acts as a virtual image sensor, acquiring Fourier spectrum images to indirectly obtain two-dimensional angle information of the light rays. An image sensor 6 is used to acquire images of the target object on the image plane, directly obtaining the two-dimensional spatial information of the image. Furthermore, this invention proposes a novel light field imaging method to address the aforementioned angle and spatial information acquisition methods.

[0026] 1) Existing light field imaging technologies based on microlens arrays use a single image sensor to simultaneously acquire two-dimensional angular and two-dimensional spatial information. This results in a trade-off between angular and spatial resolution, leading to a decrease in the spatial resolution of the image. In contrast, this invention uses two image sensors to acquire two-dimensional angular and two-dimensional spatial information separately. Therefore, the trade-off between angular and spatial resolution is eliminated. Furthermore, the image sensors directly acquire the image of the target object on the image plane to obtain two-dimensional spatial information, ensuring that the image spatial resolution matches the spatial resolution of the image sensor's full sampling.

[0027] 2) Existing methods utilize spatial light modulators to modulate on the image plane, with the spatial light modulator acting as a virtual image sensor to indirectly acquire two-dimensional spatial information of the image; and image sensors acquire Fourier spectrum images on the Fourier spectrum plane to directly obtain two-dimensional angular information. While this method achieves the same angular resolution as the image sensor (full sampling resolution), the spatial resolution of the reconstructed image is determined by the spatial modulation resolution of the spatial light modulator. To ensure the reconstructed image's spatial resolution matches the full sampling resolution of the spatial light modulator, the number of modulation and signal acquisition operations must be at least equal to the number of modulation pixels in the spatial light modulator, resulting in excessively long imaging times, which is detrimental to imaging dynamic scenes. To overcome this problem of excessively long imaging times, this invention utilizes an image sensor to directly acquire images of the target object on the image plane, obtaining two-dimensional spatial information. This ensures that the image's spatial resolution matches the full sampling resolution of the image sensor. Furthermore, by controlling the size of the macro-modulation units in the spatial light modulator via computer, the number of macro-modulation units is reduced, thereby reducing the angular resolution of the light rays and shortening the imaging time. The imaging time can be adjusted to the minimum of a single acquisition by the image sensor. Attached Figure Description

[0028] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings...

[0029] Figure 1 This is a schematic diagram of the imaging system device used in the embodiments of the present invention, attached. Figure 1The labels in the accompanying drawings are explained as follows: 1-front imaging lens group, 2-Fourier transform lens group, 3-beam splitter, 4-spatial light modulator, 5-rear imaging lens group, 6-image sensor, and 7-computer;

[0030] Figure 2 This is a Fourier transform basis pattern corresponding to one of the spatial frequencies used in the embodiments of the present invention, consisting of three different initial phases of 0, 2π / 3, and 4π / 3.

[0031] Figure 3 This is one of the pairs of Hadamard transform substrate patterns P used in the embodiments of the present invention. 2k′-1 (m p ,n p ), P 2k′ (m p ,n p ) Schematic diagram;

[0032] Figure 4 This is one of the pairs of random patterns P used in the embodiments of the present invention. 2k′-1 (m p ,n p ), P 2k′ (m p ,n p ) Schematic diagram;

[0033] Figure 5 This is the reconstructed Fourier spectrum image in this embodiment of the invention. A schematic diagram arranged according to the macro-pixel coordinates of the image sensor;

[0034] Figure 6 This is a viewpoint image reconstructed by extracting grayscale values ​​of the same pixel coordinates from each Fourier spectrum image in this embodiment of the invention: A schematic diagram arranged in the order of Fourier spectrum images;

[0035] Figure 7 This is a schematic diagram of the refocusing result using Fourier transform basis pattern modulation in an embodiment of the present invention, wherein, Figure 7 (a) Figure 7 (b) Figure 7 (c) Figure 7 (d) Figure 7 (e) and Figure 7 (f) are the first, second, third, fourth, fifth, and sixth images obtained from the refocusing position from near to far, respectively. The position indicated by the arrow in the image is the refocusing position.

[0036] Figure 8 This is a schematic diagram of the refocusing result using Hadamard transform substrate pattern modulation in an embodiment of the present invention, wherein, Figure 8 (a) Figure 8 (b) Figure 8 (c) Figure 8 (d) Figure 8 (e) and Figure 8 (f) are the first, second, third, fourth, fifth, and sixth images obtained from the refocusing position from near to far, respectively. The position indicated by the arrow in the image is the refocusing position.

[0037] Figure 9 This is a schematic diagram of the refocusing result using random pattern modulation in an embodiment of the present invention, wherein, Figure 9 (a) Figure 9 (b) Figure 9 (c) Figure 9 (d) Figure 9 (e) and Figure 9 (f) are the first, second, third, fourth, fifth and sixth images obtained from the refocusing position from near to far, respectively. The position indicated by the arrow in the image is the refocusing position. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] Example 1

[0040] This embodiment provides a light field imaging device with adjustable angular resolution and imaging time, such as... Figure 1As shown, the device includes a front imaging lens group 1, a Fourier transform lens group 2, a beam splitter 3, a spatial light modulator 4, a rear imaging lens group 5, an image sensor 6, and a computer 7. The front imaging lens group 1 is a camera lens with a focal length f1 = 50mm; the Fourier transform lens group 2 is a cemented doublet with a focal length f2 = 100mm; the beam splitter 3 is a polarizing cuboid prism with a side length of 1 inch; the spatial light modulator 4 is a silicon-based liquid crystal micro-spatial light modulator with a modulation pixel size of 8μm and a modulation pixel count of 1920×1200; the rear imaging lens group 5 is a biconvex lens with a focal length f3 = 50.8mm; and the image sensor 6 is a color CMOS camera with a pixel size of 3.45μm and a pixel count of 2448×2048. The image plane of the front imaging lens group 1 that images the target object coincides with the front focal plane of the Fourier transform lens group 2; the rear focal plane of the Fourier transform lens group 2 coincides with the modulation plane of the spatial light modulator 4; the front focal plane of the rear imaging lens group 5 coincides with the rear focal plane of the Fourier transform lens group 2; the photosensitive surface of the image sensor 6 coincides with the rear focal plane of the rear imaging lens 5; the computer 7 is connected to the spatial light modulator 4 and the image sensor 6.

[0041] Example 2

[0042] This embodiment 2 is based on the light field imaging method with adjustable angle resolution and imaging time disclosed in embodiment 1 above. The target object is composed of a doll and a pencil placed at different depths. The light field imaging method includes the following steps:

[0043] S1. Modulate the Fourier spectrum image:

[0044] In the 1920×1200 pixel spatial light modulator 4, 120×75 adjacent modulation pixels are set as a macro modulation unit, forming 16×16 macro modulation units of the same size.

[0045] Based on the principle of three-step phase-shift Fourier single-pixel imaging, the computer pre-generates 386 different Fourier transform substrate patterns. These patterns are sequentially loaded into the spatial light modulator 4 to modulate the Fourier spectrum image; where: 1 ≤ m p ≤16、1≤n p ≤16, 1≤k≤386; (m p ,n p ) represents the coordinates of the macro-modulation unit of the modulation pattern; f x f y For the spatial frequency components in the x and y directions of the modulation pattern, corresponding to each spatial frequency (f x ,f y ), initial phase Take three different values: 0, 2π / 3, and 4π / 3. Figure 2 The Fourier transform basis patterns corresponding to three different initial phases, 0, 2π / 3, and 4π / 3, for one of the spatial frequencies are shown.

[0046] Alternatively, based on the principle of Hadamard single-pixel imaging, computer 7 generates 512 different Hadamard transform substrate patterns: P1(m p ,n p P2(m) p ,n p ), ..., P k (m p ,n p ), ..., P 512 (m p ,n p These patterns are sequentially loaded into the spatial light modulator 4; these Hadamard transform basis patterns are binary patterns of 0 and 1, i.e., P k (m p ,n p ) = 0 or P k (m p ,n p ) = 1, and P 2k′ (m p ,n p )=[1-P 2k′-1 (m p ,n p )], 1≤k′≤256, k′ is an integer, 1≤k≤512, Figure 3 Showing one of the Hadama transform basis patterns P 2k′-1 (m p ,n p ), P 2k ′(m p ,n p ).

[0047] Alternatively, based on the single-pixel imaging principle of computational ghost imaging, computer 7 generates 512 different random patterns: P1(m p ,n p P2(m) p ,n p ), ..., P k (m p ,n p ), ..., P 512 (m p ,n pThese patterns are sequentially loaded into the spatial light modulator 4; these random patterns are binary patterns of 0s and 1s, i.e., P k (m p ,n p ) = 0 or P k (m p ,n p ) = 1, and P 2k′ (m p ,n p )=[1-P 2k′-1 (m p ,n p )], 1≤k′≤256, k′ is an integer, 1≤k≤512, Figure 4 One of the random patterns P is shown. 2k′-1 (m p ,n p ), P 2k′ (m p ,n p ).

[0048] S2. Target object image acquisition:

[0049] The Fourier spectrum image of the light field of the target object is modulated by the spatial light modulator 4, and then the target object is imaged onto the photosensitive surface of the image sensor 6 by the rear imaging lens group 5. The acquisition of the image sensor 6 and the modulation of the spatial light modulator 4 are synchronized.

[0050] When using 386 Fourier transform basis pattern modulations, image sensor 6 acquires 386 images of the target object modulated by spatial light modulator 4: I1(m c ,n c ), I2(m c ,n c ), ..., I k (m c ,n c ), ..., I 386 (m c ,n c ).

[0051] Alternatively, when using 512 Hadamard transform substrate patterns for modulation, image sensor 6 acquires 512 images of the target object modulated by spatial light modulator 4: I1(m c ,n c ), I2(m c ,n c ), ..., I k (m c ,n c ), ..., ...I 512 (m c ,n c ).

[0052] Alternatively, when using 512 random pattern modulations, image sensor 6 acquires 512 images of the target object modulated by spatial light modulator 4: I1(m c ,n c ), I2(m c ,n c ), ..., I k (m c ,n c ), ..., I 512 (m c ,n c Among them: (m) c ,n c () represents the pixel coordinates of the photosensitive surface of image sensor 6, which consists of 2448×2048 pixels, where 1≤m c ≤2448、1≤n c ≤2048.

[0053] Each color image acquired by image sensor 6 is decomposed into three monochrome images: red (R), green (G), and blue (B).

[0054] S3. Fourier spectral image reconstruction:

[0055] Each macropixel is defined as a group of 4×4 adjacent pixels on the photosensitive surface of image sensor 6. The grayscale value of each macropixel is the sum of the grayscale values ​​of the 4×4 pixels. Each macropixel acts as a single-pixel detector. Therefore, there are a total of 612×512 single-pixel detectors of the same size among the 2448×2048 photosensitive pixels. The following steps are performed on the three monochrome images: red (R), green (G), and blue (B).

[0056] When using 386 Fourier transform substrate patterns for modulation, the 386 gray values ​​obtained by each single-pixel detector are used to reconstruct a Fourier spectral image modulated by spatial light modulator 4 using the three-step phase-shift Fourier single-pixel imaging reconstruction method, resulting in a total of 612×512 Fourier spectral images.

[0057] Alternatively, when using 512 Hadamard transform substrate pattern modulations, the 512 grayscale values ​​obtained by each single-pixel detector are used to reconstruct a Fourier spectrum image modulated by spatial light modulator 4 using the Hadamard single-pixel imaging reconstruction method, resulting in a total of 612×512 Fourier spectrum images:

[0058] Alternatively, when using 512 random pattern modulations, for each of the 512 grayscale values ​​obtained by the single-pixel detector, a Fourier spectrum image modulated by the spatial light modulator 4 is reconstructed using the single-pixel reconstruction method of ghost imaging, resulting in a total of 612×512 Fourier spectrum images:

[0059] S4. Viewpoint Image Reconstruction:

[0060] The reconstructed 612×512 Fourier spectrum images are arranged according to the coordinates of the 6 macropixels of the image sensor, as follows: Figure 5 As shown, gray values ​​with the same pixel coordinates are extracted from each image and then rearranged according to the order of the Fourier spectrum images to form a single viewpoint image, resulting in a total of 16×16 viewpoint images: like Figure 6 As shown.

[0061] S5. Target object image refocusing and reconstruction:

[0062] Using the 16×16 viewpoint images obtained above, one viewpoint image is used as a reference to translate all other viewpoint images. The translation amount is determined by the relative position between the Fourier spectrum of each viewpoint image and the Fourier spectrum of the reference viewpoint image, as well as the refocusing distance d. All translated viewpoint images are then superimposed to obtain a single image of the target object with a refocusing distance of d. Different refocusing distances were set to obtain different refocused target object images. The three monochrome refocused images (red R, green G, and blue B) were then combined into a single color refocused image.

[0063] Figure 7 The refocusing results using Fourier transform basis pattern modulation are presented. Figure 7 (a) Figure 7 (b) Figure 7 (c) Figure 7 (d) Figure 7 (e) and Figure 7 (f) are the first, second, third, fourth, fifth and sixth images obtained from the refocusing position from near to far, respectively. The position indicated by the arrow in the image is the refocusing position.

[0064] Figure 8 The refocusing results using Hadamard transform basis pattern modulation are presented. Figure 8 (a) Figure 8 (b) Figure 8 (c) Figure 8 (d) Figure 8 (e) and Figure 8(f) are the first, second, third, fourth, fifth and sixth images obtained from the refocusing position from near to far, respectively. The position indicated by the arrow in the image is the refocusing position.

[0065] Figure 9 The refocusing results using random pattern modulation are presented. Figure 9 (a) Figure 9 (b) Figure 9 (c) Figure 9 (d) Figure 9 (e) and Figure 9 (f) are the first, second, third, fourth, fifth and sixth images obtained from the refocusing position from near to far, respectively. The position indicated by the arrow in the image is the refocusing position.

[0066] Figure 7 , Figure 8 and Figure 9 The light field imaging results show that light field imaging can be achieved by using different spatial light modulators to modulate patterns, combined with corresponding single-pixel imaging methods, and the imaging effects are similar.

[0067] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A light field imaging device with adjustable angular resolution and imaging time, characterized in that, The light field imaging device includes: a front imaging lens group (1), a Fourier transform lens group (2), a beam splitter (3), a spatial light modulator (4), a rear imaging lens group (5), an image sensor (6), and a computer (7). The image plane of the front imaging lens group (1) for imaging the target object coincides with the front focal plane of the Fourier transform lens group (2); the rear focal plane of the Fourier transform lens group (2) coincides with the modulation plane of the spatial light modulator (4); the beam splitter (3) is disposed between the Fourier transform lens group (2) and the spatial light modulator (4); the front focal plane of the rear imaging lens group (5) coincides with the rear focal plane of the Fourier transform lens group (2); the photosensitive surface of the image sensor (6) coincides with the rear focal plane of the rear imaging lens (5); and the computer (7) is connected to the spatial light modulator (4) and the image sensor (6) respectively. The light emitted by the target object first passes through the front imaging lens group (1), which images the target object onto the vicinity of the front focal plane of the Fourier transform lens group (2); then the light passes through the Fourier transform lens group (2) and the beam splitter (3) in sequence, forming a Fourier spectrum image on the rear focal plane of the Fourier transform lens group (2); the Fourier spectrum image is modulated by the spatial light modulator (4), and the modulated light is reflected back to the beam splitter (3). After passing through the beam splitter (3), the light passes through the rear imaging lens group (5), which images the target object onto the photosensitive surface of the image sensor (6), and the image of the target object is acquired by the image sensor (6); wherein, the computer (7) is used to control the modulation of the Fourier spectrum image by the spatial light modulator (4) and the synchronous acquisition of the target object image by the image sensor (6), and is used to generate the modulation pattern of the spatial light modulator (4), analyze and process the image data, reconstruct the refocusing image, display the imaging results and store the data.

2. The light field imaging device according to claim 1, characterized in that, The front imaging lens group (1) includes a focusing device, which images target objects at different distances onto the vicinity of the front focal plane of the Fourier transform lens group (2) by focusing.

3. The light field imaging device according to claim 1, characterized in that, The front imaging lens group (1) also includes a microscope objective and a tube lens, which image the small target object onto the vicinity of the front focal plane of the Fourier transform lens group (2).

4. A light field imaging method based on the light field imaging device according to any one of claims 1 to 3, characterized in that, The steps of the light field imaging method are as follows: S1. The Fourier spectrum image is modulated: The target object is imaged onto the front focal plane of the Fourier transform lens group (2) through the front imaging lens group (1), and the modulation surface of the spatial light modulator (4) is set on the rear focal plane of the Fourier transform lens group (2), wherein the number of modulation pixels of the spatial light modulator (4) is M. s ×N s M is formed under the control of computer (7). P ×N p One macro modulation unit; the computer (7) pre-generated K-amplitude spatial light modulation pattern: P1(m p ,n p P2(m) p ,n p ), ..., P k (m p ,n p ), ..., P K (m p ,n p ), which are sequentially loaded onto the spatial light modulator (4) to modulate the Fourier spectrum image sequentially, wherein P k (m p ,n p Let P be the k-th spatial light modulation pattern. k (m p ,n p The coordinates of the macro-modulation unit corresponding to the spatial light modulator (4) are (m p ,n p ), 1≤M P ≤M s , 1≤N p ≤N s , 1≤k≤K, 1≤m p ≤M P 1≤n p ≤N p , k, K, m p n p M p N p M s N s It is a positive integer; S2. Image Acquisition of Target Object: The Fourier spectrum image of the target object's light field is modulated by the spatial light modulator (4), and then imaged onto the photosensitive surface of the image sensor (6) by the rear imaging lens group (5). The acquisition of the image sensor (6) and the modulation of the spatial light modulator (4) are synchronized. The image sensor acquires K images of the target object modulated by the spatial light modulator (4): I1(m c ,n c ), I2(m c ,n c ), ..., I k (m c ,n c ), ..., I K (m c ,n c ), where: I k (m c ,n c Let I be the k-th image of the target object. k (m c ,n c The pixel coordinates corresponding to the photosensitive surface of the image sensor (6) are (m c ,n c ), 1≤m c ≤M c 1≤n c ≤N c The photosensitive surface of the image sensor (6) is made of M c ×N c Composed of pixels, m c n c M c N c It is a positive integer; S3. Fourier Spectrum Image Reconstruction: Each photosensitive pixel of the image sensor (6) is treated as a single-pixel detector. For the K gray values ​​detected by each single-pixel detector, an image reconstruction algorithm based on single-pixel imaging technology is used to reconstruct a Fourier spectrum image modulated by the spatial light modulator (4), resulting in a total of M images. c ×N c Fourier spectrum image: S4. Viewpoint Image Reconstruction: Reconstruct the M image from the above-mentioned M... c ×N c The four-dimensional images are arranged in the order of pixel coordinates of the image sensor (6). Gray values ​​of the same pixel coordinates are extracted from each image, and then rearranged in the order of the four-dimensional images to form a single viewpoint image, resulting in a total of M images. p ×N p Aperture view image: I (1,1) (m c ,n c ), I (1,2) (m c ,n c ), ... S5. Target object image refocusing and reconstruction: using the M obtained above... p ×N p Take one of the four viewpoint images as a reference and translate all the other viewpoint images. The translation amount is determined by the relative position between the Fourier spectra of each viewpoint image and the Fourier spectra of the reference viewpoint image, as well as the refocusing distance d. Then, superimpose all the translated viewpoint images to obtain a single image of the target object with a refocusing distance of d. By setting different refocusing distances, different images of the target object can be obtained.

5. The light field imaging method according to claim 4, characterized in that, In step S1, the computer (7) will connect the adjacent N in the spatial light modulator 4 ms Each modulation pixel is set as a macro modulation unit to generate M P ×N p There are N macro modulation units, and the size of each macro modulation unit is N. ms The sum of the modulation pixels, and the N of different macro-modulation units. ms Same or different, 1≤N ms ≤(M s ×N s ); where N ms It is a positive integer.

6. The light field imaging method according to claim 5, characterized in that, In step S2, the image sensor (6) is a color image sensor, which acquires K color images of the target object after being modulated by the spatial light modulator (4). Each color image is then decomposed into three monochrome images: red (R), green (G), and blue (B). The monochrome images are reconstructed using Fourier spectrum, viewpoint, and target object image refocusing according to steps S3, S4, and S5, respectively. The three images (R, G, and B) obtained at the same refocusing distance are then combined into a single color refocusing image.

7. The light field imaging method according to claim 1, characterized in that, The computer (7) pre-generated K spatial light modulation patterns are Fourier transform basis patterns. Using the image reconstruction algorithm of Fourier single-pixel imaging technology, the Fourier spectrum image of the light field of the target object is reconstructed.

8. The light field imaging method according to claim 1, characterized in that, The computer (7) pre-generated K spatial light modulation patterns are Hadamard transform basis patterns. Using the image reconstruction algorithm of Hadamard single-pixel imaging technology, the Fourier spectrum image of the light field of the target object is reconstructed.

9. The light field imaging method according to claim 1, characterized in that, The computer (7) pre-generated K spatial light modulation patterns are random patterns. Using the image reconstruction algorithm of single-pixel imaging technology of computational ghost imaging, the Fourier spectrum image of the light field of the target object is reconstructed.

10. The light field imaging method according to claim 1, characterized in that, In step S3, the N adjacent pixels in the image sensor (6) are... mc Each photosensitive pixel is set as a macro pixel, and the grayscale value of each macro pixel is N. mc The sum of the gray values ​​of each photosensitive pixel, and the N values ​​of different macropixels. mc Same or different; each macropixel acts as a single-pixel detector. Using the K grayscale values ​​detected by each single-pixel detector, and based on the image reconstruction algorithm of single-pixel imaging technology, each single-pixel detector reconstructs a Fourier spectrum image, where N... mc It is a positive integer.

Citation Information

Patent Citations

  • Light field camera and imaging method

    CN112804513A

  • Focused based depth map acquisition

    US20180104009A1