Depth of field restoration apparatus, method, device and medium based on spatial light modulator

Abstract

The application discloses a depth-of-field recovery device and method based on a spatial light modulator, equipment and a medium, the method comprising: acquiring first image data of a first light path and second image data of a second light path respectively; performing parallax calculation on the first image data and the second image data to obtain a first binocular parallax map of a sample; processing the first binocular parallax map by using a parallax principle to reconstruct a first three-dimensional topographic map of the sample; setting a depth calibration threshold range of the depth of field, and dividing a plurality of grids for aberration calibration in the first three-dimensional topographic map according to the depth calibration threshold range; completing the calibration of the field-of-view aberration by aberration calibration on the grids; reacquiring a corresponding second binocular parallax map according to the calibrated aberration; processing the second binocular parallax map by using the parallax principle until a second three-dimensional topographic map meeting the depth-of-field accuracy recovery requirement is reconstructed. The application improves the depth-of-field recovery accuracy and the depth-of-field recovery range.

Description

Technical Field

[0001] This invention relates to the field of optical imaging technology, specifically to a spatial light modulator-based depth-of-field restoration device, method, apparatus, and medium. Background Technology

[0002] In traditional techniques, two types of structures are commonly used to achieve large depth-of-field microscopic observations, both of which have certain technical limitations:

[0003] Stereo microscopy systems are based on the principle of binocular stereo microscopy. They calculate the difference in parallax angles between samples at different depths, constructing a mathematical model to calculate the depth information of the samples. Common stereo microscopy structures include Greenau-type stereo structures and CMO-type stereo structures.

[0004] Its technical limitations are twofold:

[0005] Small depth of field: The depth of field of this type of system is related to the numerical aperture of the system, that is, the high spatial resolution and the depth of field are mutually restrictive.

[0006] Defocusing of the object plane will lead to a serious deterioration of image quality: High-resolution microscope objectives have a large numerical aperture and are sensitive to spherical aberration. For samples with a certain depth, the sample area located on the object distance plane will be clearly imaged, while the other defocused areas will be blurred.

[0007] Traditional zoom measurement methods combine the limited depth of field of the optical system with precise vertical scanning technology, that is, stepwise change of the object distance to adjust the focus of objects at different depths to obtain multi-focus image sequences, and then use image processing algorithms to achieve full-focus image fusion.

[0008] Its technical limitations are twofold:

[0009] Long sample shooting time: Step-by-step adjustment of the object distance will prolong the shooting time;

[0010] High performance requirements for mechanical devices: This type of system places high demands on the precision and stability of the mechanical control system. Summary of the Invention

[0011] To overcome the shortcomings of existing stereomicroscopy systems in the background art, such as small depth of field (the depth of field of these systems is related to the numerical aperture of the system, meaning that high spatial resolution and depth of field are mutually restrictive), and object defocusing leading to severe image quality degradation (high-resolution microscope objectives have large numerical apertures and are sensitive to spherical aberration; for samples with a certain depth, the sample area located on its object distance plane will be clearly imaged, while the remaining defocused areas will be blurred), the present invention aims to provide a depth of field restoration device, method, apparatus, and medium based on a spatial light modulator.

[0012] To achieve the above objectives, the present invention adopts the following technical solution:

[0013] A first aspect of the present invention provides a depth-of-field recovery device based on a spatial light modulator, comprising a sample stage, a microscope objective, a light modulation module, a microscope tube module, and a photodetector arranged sequentially; the light modulation module includes a first spatial light modulator and a second spatial light modulator, the first and second spatial light modulators being symmetrical with respect to the central axis of the microscope objective; the microscope tube module includes a first zoom lens group and a second zoom lens group, the first and second zoom lens groups being symmetrical with respect to the central axis of the microscope objective.

[0014] The sample to be observed is placed on the sample stage. The light waves scattered by the sample are collected by a microscope objective. The microscope objective magnifies and images the collected light waves to form a first optical path and a second optical path. The first spatial light modulator is set at the entrance pupil of the first optical path. The light waves of the first optical path are converged and imaged onto the photodetector by the first spatial light modulator and the first zoom lens group in sequence. The second spatial light modulator is set at the entrance pupil of the second optical path. The light waves of the second optical path are converged and imaged onto the photodetector by the second spatial light modulator and the second zoom lens group in sequence.

[0015] The photodetector is connected to the computer.

[0016] In some possible implementations, the first spatial light modulator and the second spatial light modulator perform field aberration modulation respectively before modulating the light wave.

[0017] A second aspect of the present invention provides a depth-of-field recovery method based on a spatial light modulator, applied to the aforementioned depth-of-field recovery device based on a spatial light modulator, the depth-of-field recovery method comprising the following:

[0018] S1: Acquire the first image data of the first optical path and the second image data of the second optical path respectively;

[0019] S2: Perform disparity calculation on the first image data and the second image data to obtain the first binocular disparity map of the sample;

[0020] S3: The first binocular disparity map is processed using the principle of parallax to reconstruct the first three-dimensional topography map of the sample;

[0021] S4: Set the depth calibration threshold range for the depth of field, and divide the first three-dimensional topography into several grids for aberration calibration according to the depth calibration threshold range;

[0022] S5: Field aberration calibration is completed by calibrating the grid aberrations;

[0023] S6: Based on the calibrated aberrations, re-acquire the corresponding second binocular disparity map;

[0024] S7: Process the second binocular parallax map using the parallax principle until a second three-dimensional topography map that meets the depth-of-field accuracy recovery requirements is reconstructed.

[0025] The beneficial effects of the present invention are as follows: by constructing a first optical path and a second optical path, which are sequentially converged and imaged onto a photodetector via a first spatial light modulator and a first zoom lens group, a first binocular parallax map is obtained. By dividing the first three-dimensional topography map into a grid for aberration calibration based on the depth calibration threshold range of the depth of field, the depth recovery accuracy and depth recovery range of the depth of field are improved. Thus, a high-precision second three-dimensional topography map that meets the depth accuracy recovery requirements is output using the calibrated aberrations.

[0026] In some implementations, the calibration of field aberrations by calibrating the grid in step S5 specifically includes the following:

[0027] S41: Set the depth calibration threshold range for the depth of field, and divide the first three-dimensional topography map into several grids for aberration calibration according to the depth calibration threshold range;

[0028] S42: Obtain the calibration wavefront information corresponding to each grid based on the initial three-dimensional topography map;

[0029] S43: Solve the calibration wavefront information to obtain the defocus amount of the corresponding field of view;

[0030] S44: Load the corresponding calibration wavefront information for each grid one by one until the aberration calibration of each grid under each field of view is completed.

[0031] In some implementations, the first spatial light modulator and the second spatial light modulator are loaded to the initial phase state before performing step S1.

[0032] A third aspect of the present invention provides a depth-of-field restoration device that applies the above-described depth-of-field restoration method, the depth-of-field restoration device comprising:

[0033] Image data acquisition module: acquires first image data of the first optical path and second image data of the second optical path respectively;

[0034] First disparity map acquisition module: Performs disparity calculation on the first image data and the second image data to obtain the first binocular disparity map of the sample;

[0035] 3D Reconstruction Module: Processes the first binocular disparity map using the principle of parallax to reconstruct the first 3D topography map of the sample;

[0036] Mesh construction module: Set the depth calibration threshold range for the depth of field, and divide the first three-dimensional topography into several meshes for aberration calibration according to the depth calibration threshold range;

[0037] Aberration calibration module: calibrates field aberrations by calibrating the grid.

[0038] Second disparity map acquisition module: Based on the calibrated aberrations, re-acquire the corresponding second binocular disparity map;

[0039] 3D output module: The second binocular parallax map is processed using the principle of parallax until a second 3D topographic map that meets the requirements for depth of field accuracy restoration is reconstructed.

[0040] In some embodiments, the aberration calibration module includes:

[0041] Wavefront information acquisition unit: acquires calibration wavefront information corresponding to each grid based on the initial three-dimensional topography map;

[0042] Defocus acquisition unit: calculates the calibration wavefront information to obtain the defocus amount of the corresponding field of view;

[0043] Field aberration calibration unit: Loads the corresponding calibration wavefront information for each grid one by one until the aberration calibration of each grid under each field of view is completed.

[0044] In some implementations, the image data acquisition module loads the first spatial light modulator and the second spatial light modulator into an initial phase state before execution.

[0045] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the depth-of-field restoration method described above. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the depth-of-field restoration device based on a spatial light modulator according to an embodiment of the present invention;

[0047] Figure 2 This is a flowchart illustrating the overall steps of the depth-of-field restoration method according to an embodiment of the present invention.

[0048] Figure 3 This is a flowchart illustrating the steps involved in calibrating field aberrations according to an embodiment of the present invention.

[0049] Figure 4 This is a schematic diagram of the depth-of-field restoration device according to an embodiment of the present invention;

[0050] Figure 5 This is a schematic diagram of the aberration calibration module according to an embodiment of the present invention.

[0051] In the figure, 1 is the sample stage; 2 is the microscope objective; 3 is the first spatial light modulator; 4 is the second spatial light modulator; 5 is the first zoom lens group; 6 is the second zoom lens group; and 7 is the photodetector. Detailed Implementation

[0052] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0053] See appendix Figure 1 As shown, the depth-of-field restoration device based on a spatial light modulator in this embodiment includes a sample stage 1, a microscope objective 2, a light modulation module, a microscope tube module, and a photodetector 7 arranged sequentially. The light modulation module includes a first spatial light modulator 3 and a second spatial light modulator 4, which are symmetrical with respect to the central axis of the microscope objective 2. The microscope tube module includes a first zoom lens group 5 and a second zoom lens group 6, which are symmetrical with respect to the central axis of the microscope objective 2.

[0054] The sample to be observed is placed on the sample stage 1. The light waves scattered by the sample are collected by the microscope objective 2. The microscope objective 2 magnifies and images the collected light waves to form a first optical path (left optical path) and a second optical path (right optical path). The first spatial light modulator 3 is set at the entrance pupil of the first optical path. The light waves of the first optical path are converged and imaged onto the photodetector 7 by the first spatial light modulator 3 and the first zoom lens group 5 in sequence. The second spatial light modulator 4 is set at the entrance pupil of the second optical path. The light waves of the second optical path are converged and imaged onto the photodetector 7 by the second spatial light modulator 4 and the second zoom lens group 6 in sequence.

[0055] The photodetector 7 is connected to the computer. The photodetector 7 converts the collected light imaging signal into an electrical signal and transmits the electrical signal to the computer. The computer amplifies, filters, and processes the electrical signal to obtain an image of the sample to be observed.

[0056] In some embodiments, the first spatial light modulator 3 and the second spatial light modulator 4 perform field-of-view aberration modulation respectively before modulating the light wave. By using the first spatial light modulator 3 and the second spatial light modulator 4 to modulate and demodulate the field-of-view aberration, aberrations are further eliminated.

[0057] This embodiment provides a depth-of-field restoration method based on a spatial light modulator, applied to the aforementioned depth-of-field restoration device based on a spatial light modulator. See attached diagram. Figure 2 As shown, the depth-of-field restoration method includes the following:

[0058] S1: Acquire the first image data of the first optical path and the second image data of the second optical path respectively.

[0059] S2: Perform disparity calculation on the first image data and the second image data to obtain the first binocular disparity map of the sample.

[0060] S3: The first binocular disparity map is processed using the principle of parallax to obtain the three-dimensional point cloud data of the sample, and the first three-dimensional topography map of the sample is reconstructed based on the three-dimensional point cloud.

[0061] S4: Set the depth calibration threshold range for the depth of field, and divide the first three-dimensional topography map into several grids for aberration calibration according to the depth calibration threshold range.

[0062] The depth recovery accuracy can be improved by optionally setting the depth calibration threshold range. The depth recovery range can be improved by optionally dividing the number of grids used for aberration calibration.

[0063] S5: Field aberration calibration is completed by calibrating the grid aberrations.

[0064] S6: Based on the calibrated aberrations, re-acquire the corresponding second binocular disparity map.

[0065] S7: Process the second binocular parallax map using the parallax principle until a second three-dimensional topography map that meets the depth-of-field accuracy recovery requirements is reconstructed.

[0066] The beneficial effects of the present invention are as follows: by constructing a first optical path and a second optical path, which are sequentially converged and imaged onto a photodetector 7 via a first spatial light modulator 3 and a first zoom lens group 5, a first binocular parallax map is obtained. By dividing the first three-dimensional topography map into a grid for aberration calibration based on the depth calibration threshold range of the depth of field, the depth recovery accuracy and depth recovery range of the depth of field are improved. Thus, a high-precision second three-dimensional topography map that meets the depth accuracy recovery requirements is output using the calibrated aberrations.

[0067] In some implementations, see Appendix Figure 3 As shown, the field aberration calibration described in S5 through grid aberration calibration specifically includes the following:

[0068] S51: Obtain the calibration wavefront information corresponding to each grid based on the initial three-dimensional topography map;

[0069] S52: Solve the calibration wavefront information to obtain the defocus amount of the corresponding field of view;

[0070] S53: Load the corresponding calibration wavefront information for each grid one by one until the aberration calibration of each grid under each field of view is completed.

[0071] By loading the corresponding calibration wavefront information onto each grid point, the defocus amount of the corresponding field of view is obtained. By combining the defocus amounts of multiple field points, the field aberration is calibrated, resulting in a clear final image and improving the depth of field of the imaging system.

[0072] In some implementations, the first spatial light modulator 3 and the second spatial light modulator 4 are loaded to the initial phase state before performing step S1.

[0073] This embodiment also provides a depth-of-field restoration device, which applies the depth-of-field restoration method described above. (See attached diagram.) Figure 4 As shown, the depth-of-field restoration device includes:

[0074] Image data acquisition module: acquires first image data of the first optical path and second image data of the second optical path respectively;

[0075] First disparity map acquisition module: Performs disparity calculation on the first image data and the second image data to obtain the first binocular disparity map of the sample;

[0076] 3D Reconstruction Module: Processes the first binocular disparity map using the principle of parallax to reconstruct the first 3D topography map of the sample;

[0077] Mesh construction module: Set the depth calibration threshold range for the depth of field, and divide the first three-dimensional topography into several meshes for aberration calibration according to the depth calibration threshold range;

[0078] Aberration calibration module: The field aberration is calibrated by calibrating the grid aberration. Specifically, the field aberration is calibrated by the first spatial light modulator 3 and the second spatial light modulator 4.

[0079] Second disparity map acquisition module: Based on the calibrated aberrations, re-acquire the corresponding second binocular disparity map;

[0080] 3D output module: The second binocular parallax map is processed using the principle of parallax until a second 3D topographic map that meets the requirements for depth of field accuracy restoration is reconstructed.

[0081] In some implementations, see Appendix Figure 5 As shown, the aberration calibration module includes:

[0082] Wavefront information acquisition unit: acquires calibration wavefront information corresponding to each grid based on the initial three-dimensional topography map;

[0083] Defocus acquisition unit: calculates the calibration wavefront information to obtain the defocus amount of the corresponding field of view;

[0084] Field aberration calibration unit: Specifically, the first spatial light modulator 3 and the second spatial light modulator 4 load the corresponding calibration wavefront information for each grid one by one until the aberration calibration of each grid under each field of view is completed.

[0085] In some implementations, the first spatial light modulator 3 and the second spatial light modulator 4 are loaded into the initial phase state before the image data acquisition module is executed.

[0086] This embodiment also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the depth-of-field restoration method described above.

[0087] The storage medium stores program instructions capable of implementing all the above methods. These program instructions can be stored in the storage medium in the form of a software product, including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, or terminal devices such as computers, servers, mobile phones, and tablets.

[0088] The processor can also be called a CPU (Central Processing Unit). A processor may be an integrated circuit chip with signal processing capabilities. A processor can also be:

[0089] A DSP (Digital Signal Processor) is a processor composed of large-scale or very large-scale integrated circuit chips used to perform specific signal processing tasks. It gradually developed to meet the needs of high-speed, real-time signal processing tasks. With the development of integrated circuit technology and digital signal processing algorithms, the implementation methods of digital signal processors are constantly changing, and their processing capabilities are continuously improving and expanding.

[0090] ASIC (Application Specific Integrated Circuit) refers to an integrated circuit designed and manufactured to meet the specific requirements of a user and the needs of a specific electronic system.

[0091] FPGA (Field Programmable Gate Array) is a further development based on programmable devices such as PAL (Programmable Array Logic) and GAL (Generic Array Logic). It emerged as a semi-custom circuit in the field of Application-Specific Integrated Circuits (ASICs), solving the shortcomings of custom circuits while overcoming the limitation of the limited gate count of original programmable devices.

[0092] A general-purpose processor, which may be a microprocessor or any conventional processor.

[0093] Other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components, etc.

[0094] The above embodiments are only for illustrating the technical concept and features of the present invention. Their purpose is to enable those skilled in the art to understand the content of the present invention and implement it. They should not be used to limit the scope of protection of the present invention. All equivalent changes or modifications made in accordance with the spirit and essence of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A depth-of-field restoration method based on a spatial light modulator, applied to a depth-of-field restoration device based on a spatial light modulator, wherein the depth-of-field restoration device based on a spatial light modulator includes: The sample stage (1), microscope objective (2), light modulation module, microscope tube module, and photodetector (7) are arranged sequentially. The light modulation module includes a first spatial light modulator (3) and a second spatial light modulator (4), which are symmetrical with respect to the central axis of the microscope objective (2). The microscope tube module includes a first zoom lens group (5) and a second zoom lens group (6), which are symmetrical with respect to the central axis of the microscope objective (2). The sample to be observed is placed on the sample stage (1). The light waves scattered by the sample are collected by the microscope objective (2). The microscope objective (2) magnifies and images the collected light waves to form a first optical path and a second optical path. The first spatial light modulator (3) is set at the entrance pupil of the first optical path. The light waves of the first optical path are converged and imaged on the photodetector (7) by the first spatial light modulator (3) and the first zoom lens group (5) in sequence. The second spatial light modulator (4) is set at the entrance pupil of the second optical path. The light waves of the second optical path are converged and imaged on the photodetector (7) by the second spatial light modulator (4) and the second zoom lens group (6) in sequence. The photodetector (7) is connected to the computer; the first spatial light modulator (3) and the second spatial light modulator (4) perform field aberration modulation respectively before modulating the light wave, characterized in that: the depth-of-field recovery method includes the following: S1: Acquire the first image data of the first optical path and the second image data of the second optical path respectively; S2: Perform disparity calculation on the first image data and the second image data to obtain the first binocular disparity map of the sample; S3: The first binocular disparity map is processed using the principle of parallax to reconstruct the first three-dimensional topography map of the sample; S4: Set the depth calibration threshold range for the depth of field, and divide the first three-dimensional topography into several grids for aberration calibration according to the depth calibration threshold range; S5: Field aberration calibration is completed by calibrating the grid aberrations; S6: Based on the calibrated aberrations, re-acquire the corresponding second binocular disparity map; S7: Process the second binocular parallax map using the parallax principle until a second three-dimensional topography map that meets the depth-of-field accuracy recovery requirements is reconstructed.

2. The depth-of-field restoration method based on a spatial light modulator according to claim 1, characterized in that: The field aberration calibration described in S5, which involves calibrating the grid, specifically includes the following: S51: Obtain the calibration wavefront information corresponding to each grid based on the first three-dimensional topography image; S52: Solve the calibration wavefront information to obtain the defocus amount of the corresponding field of view; S53: Load the corresponding calibration wavefront information for each grid one by one until the aberration calibration of each grid under each field of view is completed.

3. The depth-of-field restoration method based on a spatial light modulator according to claim 1 or 2, characterized in that: Before performing step S1, the first spatial light modulator (3) and the second spatial light modulator (4) are loaded into the initial phase state.

4. A depth-of-field restoration device, characterized in that, The depth-of-field restoration method according to any one of claims 1-3, wherein the depth-of-field restoration device comprises: Image data acquisition module: acquires first image data of the first optical path and second image data of the second optical path respectively; First disparity map acquisition module: Performs disparity calculation on the first image data and the second image data to obtain the first binocular disparity map of the sample; 3D Reconstruction Module: Processes the first binocular disparity map using the principle of parallax to reconstruct the first 3D topography map of the sample; Mesh construction module: Set the depth calibration threshold range for the depth of field, and divide the first three-dimensional topography into several meshes for aberration calibration according to the depth calibration threshold range; Aberration calibration module: calibrates field aberrations by calibrating the grid. Second disparity map acquisition module: Based on the calibrated aberrations, re-acquire the corresponding second binocular disparity map; 3D output module: The second binocular parallax map is processed using the principle of parallax until a second 3D topographic map that meets the requirements for depth of field accuracy restoration is reconstructed.

5. The depth-of-field restoration device according to claim 4, characterized in that: The aberration calibration module includes: Wavefront information acquisition unit: acquires calibration wavefront information corresponding to each grid based on the first three-dimensional topography image; Defocus acquisition unit: calculates the calibration wavefront information to obtain the defocus amount of the corresponding field of view; Field aberration calibration unit: Loads the corresponding calibration wavefront information for each grid one by one until the aberration calibration of each grid under each field of view is completed.

6. The depth-of-field restoration device according to claim 4 or 5, characterized in that: Before the image data acquisition module is executed, the first spatial light modulator (3) and the second spatial light modulator (4) are loaded into the initial phase state.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the depth-of-field restoration method of claim 1.

Images