A three-dimensional reconstruction method and device based on multi-dimensional modulation and a microscopic reconstruction system

Abstract

A three-dimensional reconstruction method, apparatus, and microscopic reconstruction system based on multidimensional modulation are disclosed. The microscopic reconstruction system includes a projection unit, an imaging unit, and a motion unit. The three-dimensional reconstruction method includes controlling the motion unit to drive the sample under test to move relative to the imaging unit along a preset direction; at each displacement position, controlling the projection unit to project complementary illumination patterns and / or illumination patterns with different spatial scales onto the sample under test; after each projection, controlling the imaging unit to acquire a projected image of the sample surface; generating a sharpness map based on the projected images acquired in each projection cycle; wherein, one projection of complementary illumination patterns or illumination patterns with different spatial scales constitutes one projection cycle; and performing three-dimensional reconstruction based on the maximum pixel value at the same pixel position in all sharpness maps to obtain the three-dimensional reconstruction result of the sample under test. A three-dimensional reconstruction method that can achieve high accuracy and stability under complex sample conditions is proposed.

Description

Technical Field

[0001] This invention relates to the field of optical three-dimensional measurement and microscopic imaging technology, specifically to a three-dimensional reconstruction method, device, and microscopic reconstruction system based on multidimensional modulation. Background Technology

[0002] With the increasing demand for micro- and nano-scale structural characterization in fields such as life sciences, materials science, and microelectronics manufacturing, microscopic imaging systems are no longer limited to two-dimensional image acquisition but are gradually developing towards high-precision three-dimensional topography reconstruction. Among various microscopic three-dimensional measurement methods, the microscopic reconstruction method based on focal plane variation has attracted widespread attention due to its relatively simple system structure, lack of complex reference optical paths, and insensitivity to environmental vibrations. The basic principle of this method is: by stepping along the optical axis (Z-axis) to change the relative position of the sample and the objective lens, a series of two-dimensional images at different focal planes are acquired. Then, for each pixel in the image, the change curve of its grayscale or sharpness along the Z-axis is analyzed, and the position with the highest sharpness is determined as the object point height corresponding to that pixel, thereby reconstructing the three-dimensional topography of the entire field of view. Summary of the Invention

[0003] The present invention aims to propose a method for achieving high-precision and stable three-dimensional reconstruction under complex sample conditions.

[0004] According to a first aspect, one embodiment provides a three-dimensional reconstruction method based on multidimensional modulation, applied in a microscopic reconstruction system, the microscopic reconstruction system including a projection unit, an imaging unit, and a motion unit, the three-dimensional reconstruction method comprising:

[0005] The motion unit is controlled to drive the sample under test to move relative to the imaging unit along a preset direction;

[0006] Each time the projection unit moves to a displacement position, it controls the projection unit to project complementary lighting patterns and / or lighting patterns with different spatial scales onto the sample under test; wherein the complementary lighting patterns have different light intensity distribution characteristics at the same spatial position.

[0007] After each projection, the imaging unit is controlled to acquire a projected image of the surface of the sample to be tested;

[0008] A sharpness map is generated based on the projected images acquired in each projection cycle; wherein, completing one projection of a lighting pattern with a complementary relationship or a lighting pattern with different spatial scales constitutes a projection cycle, and the sharpness map is used to characterize the sharpness of each pixel position;

[0009] The three-dimensional reconstruction result of the sample under test is obtained by performing three-dimensional reconstruction based on the maximum pixel value at the same pixel position in all resolution images.

[0010] In some embodiments, when the projection unit projects complementary illumination patterns onto the sample under test, the projected images acquired in each projection cycle include a first complementary projection image and a second complementary projection image. Generating a sharpness map based on the projected images acquired in each projection cycle includes:

[0011] The sharpness map is calculated according to a preset first sharpness calculation formula; wherein, the first sharpness calculation formula is:

[0012]

[0013] in, Represents a resolution map. Represents the first complementary projection image. This represents the second complementary projection image.

[0014] In some embodiments, when the projection unit projects illumination patterns of different spatial scales onto the sample under test, the projected images acquired in each projection cycle include projected images at different spatial scales k. Generating a sharpness map based on the projected images acquired in each projection cycle includes:

[0015] The sharpness map is calculated according to a preset second sharpness calculation formula; wherein, the second sharpness calculation formula is:

[0016]

[0017] in, Represents a resolution map. This represents the sharpness weighting function at spatial scale k. This represents the sharpness image corresponding to the projected image at the k-th spatial scale.

[0018] In some embodiments, controlling the imaging unit to acquire a projected image of the surface of the sample to be tested after each projection includes:

[0019] After each projection, the imaging unit is controlled to acquire projection images of the surface of the sample under test according to at least two different imaging parameters, so as to obtain projection images under at least two different imaging parameters;

[0020] The step of generating a sharpness map based on the projected images acquired in each projection cycle includes:

[0021] For the illumination pattern projected in each projection cycle, the projected images obtained after each projection of the illumination pattern in the projection cycle under at least two different imaging parameters are fused using a preset image fusion formula to obtain a fused image.

[0022] A sharpness map is generated from multiple fused images in each projection cycle.

[0023] In some embodiments, the image fusion formula is expressed as follows:

[0024]

[0025] in, Indicates a fused image. This represents the weighting function for the e-th imaging parameter. This represents the projected image under the e-th imaging condition.

[0026] In some embodiments, the step of performing 3D reconstruction based on the maximum pixel value at the same pixel location in all sharpness maps to obtain the 3D reconstruction result of the sample under test includes:

[0027] For each pixel location, obtain the maximum value among all sharpness values ​​in the sharpness maps of that pixel location, obtain the sharpness value of that pixel location in the sharpness map adjacent to the sharpness map corresponding to the maximum value, obtain the adjacent sharpness value, and fit the first estimated curve based on the maximum value, the adjacent sharpness value and the corresponding displacement position;

[0028] The maximum value of the first estimated curve is taken as the grayscale result in the 3D reconstruction result of the pixel location, and the displacement position corresponding to the maximum value of the first estimated curve is taken as the depth result in the 3D reconstruction result of the pixel location.

[0029] In some embodiments, after performing 3D reconstruction based on the maximum pixel value at the same pixel location in all sharpness maps to obtain the 3D reconstruction result of the sample to be tested, the 3D reconstruction method further includes:

[0030] For each pixel position in the 3D reconstruction result of the sample to be tested, the pixels at the pixel positions whose sharpness values ​​do not meet the preset sharpness requirements are filtered out to obtain the pixels at the remaining pixel positions, and the 3D reconstruction result is regenerated based on the pixels at the remaining pixel positions.

[0031] According to a second aspect, one embodiment provides a three-dimensional reconstruction device based on multidimensional modulation, applied in a microscopic reconstruction system, the microscopic reconstruction system including a projection unit, an imaging unit, and a motion unit, the three-dimensional reconstruction device comprising:

[0032] The unit control module is used to control the motion unit to drive the sample under test to move relative to the imaging unit along a preset direction;

[0033] The pattern projection module is used to control the projection unit to project complementary lighting patterns and / or lighting patterns with different spatial scales onto the sample under test each time the module moves to a displacement position; wherein the complementary lighting patterns have different light intensity distribution characteristics at the same spatial position.

[0034] An image acquisition module is used to control the imaging unit to acquire a projected image of the surface of the sample under test after each projection.

[0035] The image processing module is used to generate a sharpness map based on the projected images acquired in each projection cycle; wherein, completing one projection of a lighting pattern with a complementary relationship or a lighting pattern with different spatial scales constitutes a projection cycle, and the sharpness map is used to characterize the sharpness of each pixel position.

[0036] The 3D reconstruction module is used to perform 3D reconstruction based on the maximum pixel value at the same pixel position in all resolution images, so as to obtain the 3D reconstruction result of the sample under test.

[0037] According to a third aspect, one embodiment provides a microscopic reconstruction system, comprising: A projection unit is used to project lighting patterns that have a complementary relationship and / or lighting patterns with different spatial scales; wherein the lighting patterns that have a complementary relationship have different light intensity distribution characteristics at the same spatial location; An imaging unit is used to acquire a projected image of the surface of the sample to be tested; A motion unit is used to drive the sample to be tested to move relative to the imaging unit along a preset direction, so that when it moves to a preset displacement position, the projection unit responds to the control command sent by the processor to project. The processor is used to process the sample under test according to the multidimensional modulation-based three-dimensional reconstruction method to obtain the three-dimensional reconstruction result of the sample under test.

[0038] According to the fourth aspect, one embodiment provides a computer program product, including a computer program and / or instructions, which, when executed by a processor, implement the multidimensional modulation-based three-dimensional reconstruction method.

[0039] According to the above embodiments of the three-dimensional reconstruction method, apparatus, microscopic reconstruction system and computer program product based on multi-dimensional modulation, the motion unit is controlled to drive the sample under test to move relative to the imaging unit along a preset direction. Since each time it moves to a displacement position, the projection unit is controlled to project a complementary lighting pattern and / or a lighting pattern with different spatial scales onto the sample under test. Complementary lighting modulation and multi-scale modulation are introduced at the lighting modulation level, so that the lighting is no longer a fixed background, but becomes an active variable participating in the focal plane determination. The projected images obtained under different lighting conditions can form a complementary and enhanced relationship in the three-dimensional reconstruction process, so that high-precision and stable three-dimensional reconstruction can still be achieved under complex sample conditions. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the structure of the microscopic reconstruction system in the embodiments of this application;

[0041] Figure 2 This is a flowchart of the three-dimensional reconstruction method based on multidimensional modulation in the embodiments of this application;

[0042] Figure 3 This is a schematic diagram of a lighting pattern with a complementary relationship in one embodiment;

[0043] Figure 4 This is a schematic diagram of lighting patterns at different spatial scales in one embodiment;

[0044] Figure 5 This is a flowchart illustrating how, in one embodiment, three-dimensional reconstruction is performed based on the maximum pixel value at the same pixel location in all resolution images to obtain the three-dimensional reconstruction result of the sample under test.

[0045] Figure 6 This is a schematic diagram of the structure of a three-dimensional reconstruction device based on multidimensional modulation in one embodiment. Detailed Implementation

[0046] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of this application. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, certain operations related to this application are not shown or described in the specification. This is to avoid obscuring the core parts of this application with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0047] Furthermore, the features, operations, or characteristics described in the specification can be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can be rearranged or adjusted in a manner obvious to those skilled in the art. Therefore, the various orders in the specification and drawings are only for the clear description of a particular embodiment and do not imply a necessary order, unless otherwise stated that a particular order must be followed.

[0048] The serial numbers assigned to components in this document, such as "first" and "second," are used only to distinguish the described objects and have no sequential or technical meaning. The terms "connection" and "linkage" used in this application, unless otherwise specified, include both direct and indirect connections (linkages).

[0049] In microscopic reconstruction methods based on focal plane variations, samples with weak surface texture and uniform reflective or transmissive properties (such as smooth silicon wafers and optical lenses) exhibit low imaging contrast, causing traditional image gradient-based sharpness evaluation functions to fail and severely reducing reconstruction accuracy. To address this, existing technologies have introduced active structured light illumination. During imaging, a pattern with a specific spatial structure (such as a checkerboard or stripes) is projected onto the sample using a digital micromirror device (DMD), artificially introducing high-frequency contrast information to enhance the sharpness differences between images at different focal planes, thereby improving the sensitivity of focal plane determination.

[0050] In existing active illumination focal plane variation microscopic reconstruction methods, a common approach is to use a single-form, fixed-parameter spatial illumination pattern, such as a regular checkerboard, stripes, or periodic dot matrix. These illumination patterns offer advantages such as well-defined spatial frequency characteristics, ease of generation, and synchronous control. In existing systems, area array imaging devices are typically used to acquire image sequences at different focal plane positions, and a driving mechanism is used to change the focal plane position. During focal plane variation, the illumination projection pattern and imaging parameters usually remain constant, and the reconstruction process is completed based on the change in image pixel sharpness under a single illumination modulation state. Sharpness is generally calculated using gradients or variance. Under ideal conditions, this method can effectively improve the discernibility of focal plane variations and reduce reconstruction errors caused by insufficient contrast in the sample itself, thus expanding the application range of focal plane variation microscopic reconstruction to some extent. However, in practical microscopic imaging and reconstruction applications, the aforementioned active illumination focal plane variation method using a fixed projection pattern still has shortcomings and limitations.

[0051] Specifically, existing technologies have at least the following shortcomings:

[0052] (1) Insufficient effective dynamic range: For samples with large spatial differences in reflectivity and scattering characteristics, the system cannot accurately capture information of the extreme bright and dark areas in a single measurement when the illumination modulation and imaging parameters are fixed, resulting in local overexposure and underexposure areas appearing simultaneously in the same field of view.

[0053] In highly reflective, overexposed areas, evaluation criteria based on image sharpness (such as gradient and variance) completely fail, leading to the loss of 3D data in that area and the appearance of "holes" or "string-like" artifacts in the reconstruction results, affecting the stability of focal plane determination. In low-reflectivity, underexposed areas, illumination of the same intensity produces a weak signal with an extremely low signal-to-noise ratio. This weak signal is overwhelmed by noise, making the sharpness calculation curve flat and the peak difficult to identify, ultimately causing a sharp decrease in reconstruction accuracy or complete failure in that area.

[0054] (2) Low reliability of sharpness evaluation operators: Traditional sharpness evaluation operators (functions) based on spatial gradients (such as Sobel, variance, etc.) or frequency domain response rely on the statistical characteristics of pixels within a local window (neighborhood) for calculation, which naturally leads to a reduction in lateral resolution during measurement. Furthermore, strong changes in out-of-focus light intensity (such as bright out-of-focus edges) themselves generate high-frequency spatial gradient signals locally, which are misjudged by the sharpness operator as "rich in detail," thus calculating a very high sharpness value at that location. This false sharpness response severely interferes with the judgment of the true focal plane position, resulting in artifacts such as protrusions or breaks in the edge region during 3D reconstruction, reducing measurement reliability.

[0055] (3) It is difficult to achieve both measurement accuracy and anti-interference capability: Existing technologies rely on fixed patterns at a single spatial scale (usually small scale), which leads to the following contradictions in terms of accuracy and anti-interference capability:

[0056] a. For small-scale patterns, such as fine checkerboard patterns, although a high theoretical lateral resolution can be achieved, the resulting sharpness response curve is extremely sensitive to noise and texture interference. In areas with rough surfaces, scratches, or stains, local textures can severely interfere with sharpness calculations, producing false peaks, leading to misjudgment of the focus position, and generating "noise artifacts."

[0057] b. For large-scale patterns, although it can improve noise resistance, it will sacrifice spatial resolution and fail to capture fine features (such as tiny scratches and precision threads), thus losing the significance of high-precision measurement.

[0058] (4) Single data processing mode and low information utilization: Existing technologies usually only utilize the original image grayscale information under a single projection state and adopt a simple sharpness maximization criterion. This single information source and simple processing mode cannot cope with complex scattering, occlusion and noise interference, which limits the upper limit of the system's performance under non-ideal conditions.

[0059] In summary, existing technologies use fixed projection patterns and imaging parameters for scanning and rely on traditional spatial resolution operators for reconstruction. These technologies suffer from a series of inherent and interconnected limitations, severely restricting their widespread application in precision industrial testing, complex materials analysis, and life science research. Therefore, there is an urgent need for a novel active illumination focal plane variation microscopy reconstruction technique that can adapt to sample characteristics and balance high dynamic range, high precision, and high robustness.

[0060] To address the aforementioned problems, this application provides a three-dimensional reconstruction method and a microscopic reconstruction system based on multidimensional modulation. Please refer to [reference needed]. Figure 1 In some embodiments, the microscopic reconstruction system 10 includes a projection unit 11, an imaging unit 12, a motion unit 13, and a processor 15. These components work together to achieve high-resolution three-dimensional microscopic imaging, which will be described in detail below.

[0061] In some embodiments, the projection unit 11 is used to project lighting patterns with complementary relationships and / or lighting patterns with different spatial scales, preferably a DMD. The complementary lighting patterns have different light intensity distribution characteristics at the same spatial location.

[0062] In some embodiments, the light emitted by the point light source 16 is converged and projected onto the projection unit 11. After modulation by the projection unit 11, it becomes a pattern conforming to a pre-designed spatial distribution. During the projection of the illumination pattern, the beam splitter 18 reflects the light towards the objective lens 17, which focuses the light onto the sample 14 to illuminate it. The light reflected from the sample 14 is then transmitted through the beam splitter 18 until it is collected by the imaging unit 12. A tube lens 19 is typically provided between the objective lens 17 and the imaging unit 12, which further focuses the signal collected by the objective lens 17 onto the imaging unit 12.

[0063] In some embodiments, the imaging unit 12 is used to acquire projected images of the surface of the sample 14 to be tested, preferably an area array camera.

[0064] In some embodiments, the motion unit 13 is used to drive the sample 14 to be tested to move relative to the imaging unit 12 along a preset direction, so that when it moves to a preset displacement position, the projection unit 11 responds to the control command sent by the processor 15 to project. The motion unit 13 drives the objective lens in the imaging unit 12 or the sample 14 to make continuous or discrete displacements along the optical axis, preferably a servo linear motor or a piezoelectric drive device.

[0065] In some embodiments, the processor 15 is used to process the sample 14 to be tested according to a three-dimensional reconstruction method based on multidimensional modulation to obtain a three-dimensional reconstruction result of the sample 14 to be tested.

[0066] The following description, in conjunction with the accompanying drawings, illustrates the three-dimensional reconstruction method based on multidimensional modulation provided in the embodiments of this application.

[0067] Figure 2 A flowchart of a three-dimensional reconstruction method based on multidimensional modulation provided in an embodiment of this application is shown, including steps S10 to S50, which are described in detail below.

[0068] Step S10: Control the motion unit 13 to drive the sample 14 to be tested to move relative to the imaging unit 12 along a preset direction.

[0069] In some embodiments, the motion unit 13 drives the sample 14 under test to make continuous or discrete displacements along a preset direction, or drives the objective lens and projection lens in the imaging unit 12 to make continuous or discrete displacements along a preset direction, or uses an inertial-free optical scanning device such as a tunable lens to realize the movement of the sample 14 under test relative to the imaging unit 12. The preset direction is the optical axis direction of the microscopic reconstruction system 10. The motion unit 13 can be a servo motor or a piezoelectric ceramic driver; servo motors or piezoelectric ceramic drivers are common methods for achieving precise axial scanning. Other alternative solutions for the motion unit 13 include voice coil motors, inertial-free optical scanning, wavefront modulation using electrically focused lenses (such as liquid lenses) or spatial light modulators to directly change the illumination focal plane optically.

[0070] Step S20: Each time the device moves to a displacement position, the control projection unit 11 projects a complementary lighting pattern and / or a lighting pattern with different spatial scales onto the sample 14 to be tested.

[0071] In some embodiments, the projection unit 11 may be a digital micromirror device, which is preferred due to its high-speed, high-contrast binary modulation characteristics. However, any optical modulation device capable of controlling and rapidly switching structured illumination patterns can be used as an alternative, such as a liquid crystal spatial light modulator, a laser scanning galvanometer system, or a programmable LED array. When moving the imaging unit 12, the displacement position is the focal plane position of the camera in the imaging unit 12; when moving the sample 14 to be tested, the position is the focal plane position of the sample 14 to be tested.

[0072] In some embodiments, illumination is no longer used as a fixed imaging background, but is introduced as an active modulation variable. Its design purpose is to provide physically directional difference information for focal plane determination. In the illumination modulation method, a complementary and multi-scale structured illumination design is proposed. That is, each time the projection unit 11 moves to a displacement position, it projects complementary illumination patterns and / or illumination patterns with different spatial scales onto the sample 14 under test, so that different modulation methods can produce significantly different light responses at corresponding spatial positions.

[0073] In some embodiments, illumination patterns with a complementary relationship have different light intensity distribution characteristics at the same spatial location. For example, in corresponding spatial regions, when one modulation scheme produces a stronger illumination response, another modulation scheme produces a weaker illumination response, or produces modulation responses with different distributions, thereby forming a distinguishable modulation pair. Please refer to... Figure 3 , Figure 3 It is a periodically distributed checkerboard pattern. Figure 3 (a) in the diagram represents the first projection pattern. , Figure 3 (b) in the diagram represents the second projection pattern. , Figure 3 (a) and (b) in the diagram represent complementary structured illumination patterns. These two patterns, designed in this way, exhibit different light intensity distribution characteristics at the same spatial location. When one pattern is bright at a certain position, the other pattern is dim at the corresponding position, thus creating complementary illumination in space and obtaining complementary light response information. The complementary modulation described above does not require strict limitation to identical focal plane positions. As long as the imaging responses acquired under different modulation states can establish a spatial correspondence, they can be used for subsequent difference analysis and focal plane determination. The complementary illumination patterns are not limited to checkerboard patterns; they can also be any pair of patterns, such as sinusoidal fringes with opposite phases, that can produce distinguishable responses to the same object point under two or more states.

[0074] In some embodiments, to balance reconstruction stability and accuracy, the projection unit 11 can project at least two structured lighting patterns of different spatial scales. Please refer to... Figure 4 , Figure 4 (a) in the diagram represents the first-scale projection pattern. Its spatial scale is relatively large, and it is used to obtain stable focal plane position information of the sample. Figure 4 (b) in the diagram represents the second-scale projection pattern. Its small spatial scale allows for the acquisition of highly sensitive focal plane position information of samples. The interleaving strategy for different spatial scales can be a fixed-period strategy or a dynamic adaptive strategy based on pre-scanning or real-time feedback.

[0075] In some embodiments, a checkerboard pattern is used as an example to illustrate "intensity complementarity" in relation to lighting patterns with complementary relationships. The essence of complementarity is to produce distinguishable, focus-dependent response differences for the same object point under two lighting conditions. Therefore, the following solutions are all within the scope of protection of this invention:

[0076] a. Anti-phase fringes: Two sets of sinusoidal or binary fringes with a phase difference of π are used;

[0077] b. Complementary polarization states: The projections have identical spatial patterns with orthogonal polarization states (e.g., perpendicular linear polarization or left-handed / right-handed circular polarization). The polarization response difference of the sample can be separated using an analyzer in front of the camera; this difference is sensitive to surface tilt, birefringence, or focusing conditions.

[0078] c. Wavelength (spectral) complementarity: Using illumination light with different center wavelengths (such as red and blue light) to project the same spatial pattern. By utilizing the chromatic aberration of the optical system or the spectral absorption differences of the sample itself, the normalized difference of the multispectral image can be calculated, which can enhance the sensitivity to specific materials or focusing states;

[0079] d. Spatiotemporal coding complementarity: Not limited to two patterns, three or more patterns projected according to a specific coding sequence (such as Gray code, sinusoidal phase shift) can be used. A "modulation contrast" or "phase" signal that is highly correlated with the ideal focus state can be extracted by the decoding algorithm. This signal also has the characteristics of pixel-level independence and anti-texture interference.

[0080] In some embodiments, regarding lighting patterns at different spatial scales: a) Number of scales: Not only can two scales (large and small) be used, but three or more scales (such as extra-large, medium, and small) can also be used to form a more refined hierarchical information acquisition system. b) Interlacing strategy: The embodiments implicitly employ sequential interlacing projection. Alternative solutions include: content-based dynamic interlacing (first, quickly pre-scan to obtain coarse-scale information, then intelligently plan the density and area of ​​fine-scale projection); regionalized interlacing (applying different scale projection strategies to different regions of interest).

[0081] In some embodiments, when the projection unit 11 projects complementary lighting patterns and / or lighting patterns with different spatial scales onto the sample 14 to be tested, the lighting patterns can be applied to the surface of the sample 14 to be tested in any of the following ways:

[0082] 1. Project sequentially within different acquisition cycles: For example, project one of the lighting patterns with complementary relationships and one of the lighting patterns with different spatial scales at the same displacement position in different acquisition cycles.

[0083] 2. Combined projection or superimposed projection within the same acquisition cycle: For example, combined projection refers to projecting complementary lighting patterns and lighting patterns with different spatial scales sequentially within one acquisition cycle, while superimposed projection refers to projecting complementary lighting patterns and lighting patterns with different spatial scales simultaneously at the same displacement position.

[0084] Through the above method, the microscopic imaging system 10 can simultaneously or time-divisionally acquire focal plane response information of complementary spatial scale features. It can be known that the expression for the illumination pattern projected by the projection unit 11 is:

[0085]

[0086] Where x and y are the pixel row index and pixel column index, respectively, and k is the side length of the checkerboard squares, which determines the size of the spatial scale. max This indicates that the current pattern is in a bright projection state.

[0087] Step S30: After each projection, control the imaging unit 12 to acquire the projected image of the surface of the sample 14 to be tested.

[0088] Step S40: Generate a sharpness map based on the projected images acquired in each projection cycle.

[0089] In some embodiments, a projection cycle is defined as one projection of complementary lighting patterns or lighting patterns with different spatial scales. A sharpness map is generated based on the projected images acquired in each projection cycle. The sharpness map can be generated using gradient-based methods, such as the Sobel operator or the Laplacian operator. The sharpness map is used to characterize the sharpness of each pixel position.

[0090] Step S50: Perform 3D reconstruction based on the maximum pixel value at the same pixel location in all resolution images to obtain the 3D reconstruction result of the sample to be tested.

[0091] In some embodiments, the maximum pixel value at the same pixel location in all resolution images is used as the key feature, and 3D reconstruction is performed in combination with techniques such as depth estimation to obtain the 3D reconstruction result. The 3D reconstruction result includes the reconstructed depth result and the reconstructed grayscale result.

[0092] In some embodiments, when the control projection unit 11 projects complementary lighting patterns onto the sample 14, the projected image acquired in each projection cycle includes a first complementary projection image and a second complementary projection image. A sharpness map is generated based on the projected image acquired in each projection cycle, including:

[0093] The sharpness map is calculated according to a preset first sharpness calculation formula; wherein, the first sharpness calculation formula is:

[0094]

[0095] in, Represents a resolution map. Represents the first complementary projection image. This represents the second complementary projection image.

[0096] In some embodiments, during the aforementioned focal plane scanning process, complementary projection patterns are projected onto sample areas at the same focal plane location. Complementarity means that, ideally, the two images produce significantly different or even opposite intensity modulation effects at the same pixel location. Under this premise, a first sharpness calculation formula is defined, which can also be called a sharpness evaluation operator. The term "sharpness evaluation operator" refers to a function relating to the difference in pixel values ​​obtained at the same pixel location under complementary projection pattern illumination. Its basic physical meaning is that if the pixel's location is in accurate focus, the imaging system exhibits high fidelity in responding to the modulation differences between the two complementary patterns. In this case, the grayscale difference can be equivalently considered as the response to energy changes before and after the pinhole in the confocal system, resulting in a large grayscale difference at that pixel location. If the pixel is out of focus, the two complementary patterns are optically blurred and "averaged" during imaging, significantly reducing the grayscale difference of the corresponding pixel. Therefore, the sharpness evaluation operator enhances the system's ability to discriminate focal plane positions and directly reflects the degree of focus of the pixel at the current focal plane position. This sharpness operator is pixel-level defined, does not rely on spatial neighborhood calculations, and does not require windowing or filtering operations. This preserves the imaging system's lateral resolution during focal plane determination and significantly reduces interference from sample texture and noise.

[0097] In some embodiments, a novel sharpness evaluation operator based on complementary projection pixel differences is proposed, significantly improving the reliability of focal plane determination. Existing technologies often rely on sharpness operators based on spatial gradients or local statistics, which are easily affected by noise, out-of-focus bright edges, and the texture of the sample itself, generating false sharpness peaks and leading to misjudgment of the focal plane. The sharpness operator constructed in this invention relies only on the response differences of the same pixel under complementary projection illumination conditions, without involving spatial neighborhood operations, thus avoiding the misjudgment problems caused by local window calculations from the imaging mechanism perspective. The true focal plane position exhibits a stable differential enhancement effect under different complementary illumination conditions, making the peak of the focal plane response curve clear and stable, thereby significantly improving the accuracy and consistency of focal plane determination. It maintains pixel-level lateral resolution, avoiding the spatial resolution degradation introduced by traditional sharpness operators. Traditional sharpness evaluation methods require statistical or filtering operations within a local window, inevitably introducing the problem of reduced lateral resolution. The sharpness operator of this invention is calculated based on the response comparison of the same pixel under different illumination modulation conditions, without the involvement of spatial neighborhoods. It completely preserves the original pixel resolution capability of the imaging system during the focal plane determination process, making the reconstruction results superior to existing technologies in terms of lateral detail.

[0098] In some embodiments, the sharpness evaluation operator is most directly represented by the absolute value of the difference, while other forms include normalized difference, squared difference, statistically based difference, etc.

[0099] In some embodiments, when the control projection unit 11 projects illumination patterns of different spatial scales onto the sample 14 to be tested, the projected images acquired in each projection cycle include projected images at different spatial scales k. A sharpness map is generated based on the projected images acquired in each projection cycle, including:

[0100] The sharpness map is calculated according to a preset second sharpness calculation formula; wherein, the second sharpness calculation formula is:

[0101]

[0102] in, Represents a resolution map. This represents the sharpness weighting function at spatial scale k. This represents the sharpness image corresponding to the projected image at the k-th spatial scale.

[0103] In some embodiments, projection units 11 project projection patterns of different spatial scales onto the sample 14 under test to enhance the system's response capability to different spatial frequency structures. For the pixel sharpness responses corresponding to projection patterns of different spatial scales, a weighted fusion mechanism is introduced, enabling the generated sharpness map to possess the high spatial resolution of small-scale patterns while retaining the high signal-to-noise ratio and anti-interference characteristics of large-scale patterns. Specifically, the sharpness weight function set in the second sharpness calculation formula ensures that the sharpness differences in rough and complex areas are mainly contributed by the sharpness under large-scale images, while the sharpness of smooth and flat areas is mainly contributed by the sharpness under small-scale images. This results in a stable and comparable sharpness map, effectively improving the system's adaptability to regions with different reflective characteristics.

[0104] In some embodiments, the weighting function The setting is crucial. In addition to being related to regional characteristics in the above embodiments, it can also be based on the quality of the sharpness curve, such as being proportional to the peak sharpness of the sharpness curve at that scale, such as kurtosis, or the signal-to-noise ratio, or based on the confidence level of the depth value, etc.

[0105] In some embodiments, to balance high measurement accuracy and high anti-interference capability, and overcome the inherent limitations of single-scale illumination, existing active structured light focal plane variation methods typically employ projection patterns with a fixed spatial scale, making it difficult to balance measurement accuracy and noise resistance. This invention introduces multi-scale structured light projection and weighted fusion of response information from different scales, enabling the system to maintain high sensitivity to fine structures while significantly improving its ability to suppress noise, surface roughness, and random textures. This allows for high-precision and stable 3D topography reconstruction even under complex sample conditions.

[0106] In some embodiments, after each projection, the imaging unit 12 is controlled to acquire a projected image of the surface of the sample 14 to be tested, including:

[0107] After each projection, the imaging unit 12 is controlled to acquire projection images of the surface of the sample 14 under at least two different imaging parameters, so as to obtain projection images under at least two different imaging parameters.

[0108] In some embodiments, a sharpness map is generated based on the projected images acquired in each projection cycle, including:

[0109] For the illumination pattern projected in each projection cycle, the projected images obtained after each projection of the illumination pattern in the projection cycle under at least two different imaging parameters are fused using a preset image fusion formula to obtain a fused image.

[0110] A sharpness map is generated from multiple fused images in each projection cycle.

[0111] In some embodiments, the expression for the image fusion formula is:

[0112]

[0113] in, Indicates a fused image. This represents the weighting function for the e-th imaging parameter. This represents the projected image under the e-th imaging condition.

[0114] In some embodiments, assuming that the imaging parameter in the above image fusion formula is the exposure time, the formula is used to fuse images obtained under different exposure parameters at the same focal plane position of the same projection pattern to obtain a fused image with extended dynamic range. The setting of the weight function makes the difference in bright areas mainly contributed by short exposure and the difference in dark areas mainly contributed by long exposure, thereby obtaining a stable and comparable brightness response and effectively improving the system's adaptability to areas with different reflective characteristics.

[0115] In some embodiments, the imaging unit 12 acquires images according to at least two different imaging parameters, for example, images under the first imaging parameter. To suppress saturation in areas of high reflectivity or strong signal, preferably, this can be achieved by setting a lower exposure time, as shown in the image under the second imaging parameter. For enhancing the signal response in weakly reflective areas, a higher exposure time can be preferably achieved, where i∈{1,2} represents the complementary pattern number. Imaging parameters include, but are not limited to, camera exposure time, camera gain, light source intensity, or equivalent imaging energy control methods, with the aim of covering both strong and weakly reflective areas that may exist on the sample surface. By obtaining multiple imaging results with different brightness distributions under different imaging parameters for the same projection pattern during the focal plane variation scanning process, subsequent data processing allows for the selection or fusion of different images to obtain effective imaging information with high signal-to-noise ratio and no saturation, thereby expanding the overall dynamic range.

[0116] In some embodiments, acquiring images using at least two different imaging parameters significantly improves the system's effective dynamic range and avoids information loss caused by overexposure and underexposure. Existing focal plane variation microscopic reconstruction methods typically employ a single exposure parameter and fixed illumination, making it difficult to simultaneously capture high-reflectivity and low-reflectivity regions within the same field of view, easily resulting in holes or severely noisy areas in the reconstruction results. This invention introduces multiple exposure parameters at the same focal plane position and performs pixel-level weighted synthesis of the multi-exposure images, enabling regions with different reflectivity characteristics to obtain effective imaging responses. This expands the system's effective dynamic range without increasing the number of scans, ensuring the integrity and continuity of three-dimensional data within the field of view.

[0117] In some embodiments, alternative solutions for multi-imaging parameter acquisition and fusion include: a) imaging parameter expansion. Besides adjusting the camera exposure time, expanding the dynamic range can also be achieved individually or in combination by adjusting the light source intensity, adjusting the camera gain, switching filters with different attenuation coefficients, etc.

[0118] b. Replacement of the HDR fusion algorithm. The example provides a framework for weighted fusion. Specific weighting functions... There are several options, such as weights based on saturation. The value is set to 1 when the pixel is unsaturated and 0 when it is saturated. The weight is based on the signal-to-noise ratio and is proportional to the local signal-to-noise ratio estimate of the image at that exposure.

[0119] c. Timing of image fusion. The above embodiment adopts the approach of "fusion of images first, then calculation of sharpness". Other alternatives include "calculation first, then fusion", that is, performing a complete sharpness calculation and focal plane localization first to obtain multiple depth maps, and then performing confidence-weighted fusion of these depth maps.

[0120] In some embodiments, various structured illumination modulations, such as projecting complementary illumination patterns, projecting illumination patterns of different spatial scales, and acquiring images according to two different imaging parameters, do not strictly require the same absolute physical location, nor do they require that all types of modulation be applied completely and simultaneously in each scan. Depending on the specific application requirements, the projection unit 11 can flexibly select single or combined modulation methods, for example, projecting only complementary illumination patterns, projecting only illumination patterns of different spatial scales, or combining multi-exposure acquisition.

[0121] Please refer to Figure 5 In some embodiments, step S50: performing three-dimensional reconstruction based on the maximum value of the pixel value at the same pixel position in all resolution images to obtain the three-dimensional reconstruction result of the sample to be tested, including steps S51 to S52, which are described in detail below.

[0122] Step S51: For each pixel location, obtain the maximum value among all sharpness values ​​of the sharpness maps at that pixel location, obtain the sharpness value of the pixel location in the sharpness map adjacent to the sharpness map corresponding to the maximum value, obtain the adjacent sharpness value, and fit the first estimated curve based on the maximum value, the adjacent sharpness value and the corresponding displacement position.

[0123] Step S52: Take the maximum value of the first estimated curve as the grayscale result in the 3D reconstruction result of the pixel location, and take the displacement position corresponding to the maximum value of the first estimated curve as the depth result in the 3D reconstruction result of the pixel location.

[0124] In some embodiments, for each pixel location, a first estimation curve is constructed based on the maximum sharpness value among all sharpness values ​​in the sharpness maps of that pixel location and its corresponding displacement position, and the grayscale and depth results are determined. To improve the accuracy of depth calculation, a local curve fitting method is employed, which performs a quadratic polynomial fitting on the sharpness values ​​in the neighborhood of the maximum value. Specifically, for any pixel location (x, y), it is assumed that the maximum sharpness value determined during the focal plane search is located at displacement position z. i To achieve higher accuracy, three consecutive displacement positions z within the neighborhood of the maximum value of this point are selected. i－1 z i and z i+1 Clarity value on , and And construct a quadratic polynomial function. Where a, b, and c are undetermined coefficients, z i This represents the depth estimate in the Z direction, z i－1 and z i+1 Indicates the relationship with z in the Z directioni Adjacent depth estimates. By solving the above quadratic polynomial function using the least squares method, the values ​​of coefficients a, b, and c can be obtained, and then the extreme values ​​of the function can be calculated. and extreme values In mathematics, extreme position It can be done through formula The calculated depth result is the reconstructed depth at pixel position (x, y). This represents the grayscale result of the reconstructed pixel position (x, y). The accuracy of the above calculation method can reach the sub-pixel level. This method can not only significantly improve the accuracy of depth calculation, but also suppress the influence of noise to a certain extent, thereby improving the overall quality of 3D reconstruction.

[0125] In some embodiments, local curve fitting is an effective subpixel localization method; other alternative algorithms include Gaussian fitting and centroid methods.

[0126] In some embodiments, after performing 3D reconstruction based on the maximum pixel value at the same pixel location in all sharpness maps to obtain the 3D reconstruction result of the sample 14 to be tested, the 3D reconstruction method further includes:

[0127] For each pixel position in the 3D reconstruction result of the sample to be tested, the pixels at the pixel positions whose sharpness values ​​do not meet the preset sharpness requirements are filtered out to obtain the pixels at the remaining pixel positions, and the 3D reconstruction result is regenerated based on the pixels at the remaining pixel positions.

[0128] In some embodiments, during 3D reconstruction, pixels are filtered out according to preset clarity requirements to ensure that the clarity of the remaining pixels meets the requirements, and the 3D reconstruction result is regenerated based on the remaining pixels to obtain a more accurate 3D reconstruction result.

[0129] Furthermore, the data processing steps and sequences described in this invention, such as multi-exposure image synthesis, cross-projection sharpness operator calculation, multi-scale sharpness fusion, and focal plane search and positioning, are only examples for reference. In actual implementation, the sequence can be adjusted or changed according to system configuration or application scenario without deviating from the core technical idea of ​​this invention, namely, to achieve pixel-level focal plane determination through illumination modulation response differences and to complete high-precision three-dimensional shape reconstruction through optional data fusion strategies.

[0130] The embodiments of this invention are flexible and applicable, allowing for free combination of illumination modulation methods, acquisition parameters, and processing sequences according to different sample characteristics, imaging conditions, or accuracy requirements. Any equivalent substitutions or improvements made within the concept and principles of this invention should be included within the scope of protection of this invention.

[0131] The embodiments of this application utilize richer dimensions of information and exhibit greater adaptability to non-ideal imaging conditions. Existing technologies typically rely on information from a single image or modulation state for focal plane determination, resulting in low information redundancy and strong dependence on imaging conditions. This invention fully leverages the multi-dimensional imaging information introduced by multiple exposures, multiple scales, and complementary projections, and constructs a robust focal plane determination mechanism through weighted fusion and differential analysis. This enables the system to operate stably under conditions of strong reflection, weak reflection, complex scattering, and noise interference, significantly broadening the applicability of focal plane variation microscopic three-dimensional reconstruction methods. The system structure and implementation complexity are moderate, facilitating engineering implementation and widespread application. This invention does not require the introduction of complex interference optical paths or high-precision reference arms; significant performance improvements can be achieved simply by improving the illumination strategy and data processing methods of existing active structured light microscopy systems, demonstrating excellent engineering feasibility and application promotion value. In summary, compared with existing technologies, this invention achieves a synergistic improvement in dynamic range, measurement accuracy, and robustness of focal plane variation microscopic 3D reconstruction without increasing system complexity by using multiple exposure parameters, multi-scale complementary structured light illumination, and a sharpness evaluation mechanism based on pixel response differences. It can effectively overcome the inherent limitations of existing methods under complex sample conditions.

[0132] In this embodiment, the core process of the invention can be summarized as follows: "During axial scanning, active projection can generate multi-state illumination patterns with pixel-level distinguishable responses. The focus level is directly evaluated by calculating the response differences of the pixels themselves in multiple states, and can be optionally optimized by combining multi-scale and multi-exposure strategies." Any technical solution that implements this core process, regardless of its specific pattern form (checkerboard, stripes, dot matrix, etc.), complementary dimensions (intensity, polarization, spectrum, etc.), scale interleaving strategy, or algorithm details, as long as it utilizes the "multi-state response difference of the pixels themselves" as the fundamental source of the focal plane criterion, should fall within the protection scope of this invention.

[0133] To address the limitations of existing active structured light illumination focal plane change microscopic 3D reconstruction methods, such as limited dynamic range, distorted sharpness criteria, the trade-off between measurement accuracy and robustness, and low information utilization, this application aims to fundamentally innovate the illumination and reconstruction paradigm and solve the following key technical challenges:

[0134] (1) To address the problem of achieving stable three-dimensional measurement with high dynamic range when there are significant spatial differences in the reflectivity and scattering characteristics of the sample surface. Existing technologies, due to the use of fixed illumination patterns and fixed imaging parameters, cannot simultaneously ensure effective imaging of both high-reflectivity and low-reflectivity areas, resulting in overexposure and underexposure within the same field of view, and missing three-dimensional data. This invention aims to solve the problem of how to adaptively expand the effective measurement dynamic range of the system during a single focal plane scan, ensuring that reliable focal plane response information can be obtained for regions with different reflectivity characteristics.

[0135] (2) To address the problem that traditional sharpness evaluation operators are highly sensitive to out-of-focus bright edges and noise, and are prone to producing false focus responses. Existing sharpness operators based on gradients or local statistical characteristics are easily affected by strong light edges, scattered light, and random noise under out-of-focus conditions, producing non-true sharpness peaks and leading to misjudgment of the focal plane. This invention aims to solve the problem of how to construct a focal plane determination mechanism that is more strongly correlated with the true imaging focal plane and is insensitive to non-ideal light intensity changes and noise interference, thereby improving the reliability and consistency of 3D reconstruction.

[0136] (3) Overcoming the technical bottleneck of balancing measurement accuracy and anti-interference capability in single-scale illumination patterns. Fixed-scale structured light illumination cannot simultaneously meet the requirements of high-resolution measurement and high robustness. This invention needs to solve how to introduce multi-scale or multi-modulation dimension information without sacrificing spatial resolution, so that the system can ensure measurement accuracy and have sufficient anti-interference capability in the presence of complex surface textures, noise interference and fine structures.

[0137] (4) To address the problem of limited system performance caused by the single information utilization and simple reconstruction strategy in the existing 3D reconstruction process. Existing technologies usually only use a single projection state and a single sharpness curve for focal plane determination, without fully exploring the multi-dimensional modulation information introduced by active illumination. This invention aims to solve how to make full use of imaging information under different illumination modulation states, and improve the stability and reconstruction accuracy of focal plane determination through a more reasonable data fusion and determination strategy, so as to adapt to complex imaging environments and multiple types of samples.

[0138] In summary, the purpose of this invention is to provide a complete and systematic solution to overcome the multiple limitations of existing technologies from principles to engineering, and ultimately to realize an active illumination microscopy method and system that can perform high dynamic range, high precision, and high efficiency three-dimensional morphology reconstruction of complex samples in harsh industrial environments.

[0139] Overall, this invention does not involve local optimization of a single algorithm step or parameter in existing active illumination focal plane change microscopic reconstruction methods. Instead, it introduces fundamental changes simultaneously at three levels: illumination modulation method, focal plane determination mechanism, and data fusion paradigm, thereby forming a set of mutually coupled and inseparable overall technical solutions.

[0140] The core of this invention is not about "how to calculate a larger sharpness value", but about redefining "what kind of information truly reflects the focus state" and building a matching lighting, acquisition and data processing mechanism around this definition.

[0141] (1) The focal plane determination is changed from "spatial structural sharpness" to "the ability to transmit illumination modulation differences".

[0142] Existing methods for microscopic reconstruction of focal plane changes generally base focal plane determination on the sharpness of the image's spatial structure, that is, inferring whether it is in focus by judging whether there are sharp edges, high-frequency textures, or grayscale variations in the image. This determination method is essentially a passive determination mechanism driven by the image content of the sample itself, and is highly sensitive to sample texture, reflectivity, and noise. This invention, however, starts from the imaging physics mechanism and transforms the basis for focal plane determination into the imaging system's ability to faithfully transmit differences in illumination modulation under the current focal plane state.

[0143] Specifically, this invention is not concerned with "whether the image is clear," but rather with whether the response differences of the same spatial location under different illumination modulation conditions can be effectively distinguished and preserved by the imaging system. When the spatial location is in true focus, the spatial differences between illumination modulations can be transmitted to the imaging result with high fidelity; while in defocus, these modulation differences are significantly weakened by the optical blurring process.

[0144] This focal plane determination mechanism realizes a fundamental shift from "image content driven" to "modulation response driven," and is the core and indispensable technical concept of this invention.

[0145] (2) Combination of complementary projection patterns and pixel-level differential sharpness operators.

[0146] Based on the above-mentioned determination mechanism, this invention further proposes a novel sharpness evaluation method, the core of which lies in the collaborative design of complementary illumination modulation and pixel-level response differential.

[0147] At the lighting level, the present invention does not use arbitrary structured light patterns, but rather structured lighting patterns that have significant complementary relationships at corresponding spatial locations, so that different patterns form discernible modulation pairs at the same spatial location.

[0148] At the level of sharpness evaluation, this invention no longer performs spatial gradient, variance or frequency domain analysis on a single image. Instead, it performs cross-state difference or equivalent difference calculation on the imaging response under different illumination modulation states at the same pixel position, thereby forming a "cross-illumination state" sharpness definition.

[0149] This sharpness operator has the following essential characteristics:

[0150] a. It does not rely on spatial neighborhood calculations, avoiding the resolution trade-off caused by window size selection;

[0151] b. Allows each pixel to independently determine the focal plane, naturally maintaining the pixel-level lateral resolution capability of the imaging system;

[0152] c. It is significantly less sensitive to interference from natural surface textures, random noise, and defocused edges, exhibiting stronger robustness.

[0153] (3) A three-in-one strategy of complementary projection, multi-scale interweaving, and multi-exposure fusion.

[0154] This invention does not introduce complementary projection or novel sharpness operators in isolation, but rather further integrates them with multi-scale modulation and multi-exposure acquisition mechanisms to form a complete and self-consistent data utilization paradigm. Specifically, the multi-exposure fusion mechanism expands the effective dynamic range at the signal strength level, avoiding response distortion caused by local overexposure or underexposure. Complementary projection modulation provides a reliable differential information source to reflect the focal plane state, while multi-scale illumination modulation balances measurement accuracy and anti-interference capability at the spatial frequency level.

[0155] The three data points are not simply superimposed, but rather integrated at the pixel level through weighted fusion. This normalizes data from different exposure conditions and scales into physically consistent and comparable equivalent response data before entering the sharpness evaluation stage. Consequently, the problems of "exposure failure" and "scale failure" are effectively avoided in the sharpness calculation stage, significantly improving the stability of the final focal plane determination and 3D reconstruction.

[0156] Please refer to Figure 6 One embodiment provides a three-dimensional reconstruction device based on multidimensional modulation, applied in a microscopic reconstruction system 10. The microscopic reconstruction system 10 includes a projection unit 11, an imaging unit 12, and a motion unit 13. The three-dimensional reconstruction device includes:

[0157] Unit control module 100 is used to control motion unit 13 to drive the sample 14 to be tested to move relative to imaging unit 12 along a preset direction;

[0158] The pattern projection module 200 is used to control the projection unit 11 to project complementary lighting patterns and / or lighting patterns with different spatial scales onto the sample 14 to be tested each time it moves to a displacement position; wherein the complementary lighting patterns have different light intensity distribution characteristics at the same spatial position.

[0159] The image acquisition module 300 is used to control the imaging unit 12 to acquire the projected image of the surface of the sample 14 after each projection.

[0160] The image processing module 400 is used to generate a sharpness map based on the projected images acquired in each projection cycle; wherein, completing one projection of a lighting pattern with a complementary relationship or a lighting pattern with different spatial scales is a projection cycle, and the sharpness map is used to characterize the sharpness of each pixel position.

[0161] The 3D reconstruction module 500 is used to perform 3D reconstruction based on the maximum pixel value at the same pixel position in all resolution images to obtain the 3D reconstruction result of the sample 14 to be tested.

[0162] One embodiment provides a computer program product, including a computer program and / or instructions, which, when executed by a processor 15, implement a three-dimensional reconstruction method based on multidimensional modulation.

[0163] Those skilled in the art will understand that all or part of the functions of the various methods in the above embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, which may include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to achieve the above functions. For example, the program can be stored in the memory of a device, and when the program in the memory is executed by the processor, all or part of the above functions can be achieved. In addition, when all or part of the functions in the above embodiments are implemented by computer programs, the program can also be stored in a server, another computer, disk, optical disk, flash drive, or external hard drive, etc., and can be downloaded or copied to the memory of a local device, or the system of the local device can be updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be achieved.

[0164] The above examples illustrate the present invention only to aid in understanding it and are not intended to limit the scope of the invention. Those skilled in the art can make various simple deductions, modifications, or substitutions based on the principles of this invention.

Claims

1. A three-dimensional reconstruction method based on multidimensional modulation, applied in a microscopic reconstruction system, the microscopic reconstruction system comprising a projection unit, an imaging unit, and a motion unit, characterized in that, The three-dimensional reconstruction method includes: The motion unit is controlled to drive the sample under test to move relative to the imaging unit along a preset direction; Each time the projection unit moves to a displacement position, it controls the projection unit to project complementary lighting patterns and / or lighting patterns with different spatial scales onto the sample under test; wherein the complementary lighting patterns have different light intensity distribution characteristics at the same spatial position. After each projection, the imaging unit is controlled to acquire a projected image of the surface of the sample to be tested; A sharpness map is generated based on the projected images acquired in each projection cycle; wherein, completing one projection of a lighting pattern with a complementary relationship or a lighting pattern with different spatial scales constitutes a projection cycle, and the sharpness map is used to characterize the sharpness of each pixel position; The three-dimensional reconstruction result of the sample under test is obtained by performing three-dimensional reconstruction based on the maximum pixel value at the same pixel position in all resolution images.

2. The three-dimensional reconstruction method as described in claim 1, characterized in that, When the projection unit projects complementary lighting patterns onto the sample under test, the projected images acquired in each projection cycle include a first complementary projection image and a second complementary projection image. Generating a sharpness map based on the projected images acquired in each projection cycle includes: The sharpness map is calculated according to a preset first sharpness calculation formula; wherein, the first sharpness calculation formula is: in, Represents a resolution map. Represents the first complementary projection image. This represents the second complementary projection image.

3. The three-dimensional reconstruction method as described in claim 1, characterized in that, When the projection unit projects illumination patterns of different spatial scales onto the sample under test, the projected images acquired in each projection cycle include projected images at different spatial scales k. Generating a sharpness map based on the projected images acquired in each projection cycle includes: The sharpness map is calculated according to a preset second sharpness calculation formula; wherein, the second sharpness calculation formula is: in, Represents a resolution map. This represents the sharpness weighting function at spatial scale k. This represents the sharpness image corresponding to the projected image at the k-th spatial scale.

4. The three-dimensional reconstruction method as described in claim 1, characterized in that, The step of controlling the imaging unit to acquire a projected image of the surface of the sample under test after each projection includes: After each projection, the imaging unit is controlled to acquire projection images of the surface of the sample under test according to at least two different imaging parameters, so as to obtain projection images under at least two different imaging parameters; The step of generating a sharpness map based on the projected images acquired in each projection cycle includes: For the illumination pattern projected in each projection cycle, the projected images obtained after each projection of the illumination pattern in the projection cycle under at least two different imaging parameters are fused using a preset image fusion formula to obtain a fused image. A sharpness map is generated from multiple fused images in each projection cycle.

5. The three-dimensional reconstruction method as described in claim 4, characterized in that, The expression for the image fusion formula is: in, Indicates a fused image. This represents the weighting function for the e-th imaging parameter. This represents the projected image under the e-th imaging condition.

6. The three-dimensional reconstruction method as described in claim 1, characterized in that, The step of performing 3D reconstruction based on the maximum pixel value at the same pixel location in all resolution images to obtain the 3D reconstruction result of the sample under test includes: For each pixel location, obtain the maximum value among all sharpness values ​​in the sharpness maps of that pixel location, obtain the sharpness value of that pixel location in the sharpness map adjacent to the sharpness map corresponding to the maximum value, obtain the adjacent sharpness value, and fit the first estimated curve based on the maximum value, the adjacent sharpness value and the corresponding displacement position; The maximum value of the first estimated curve is taken as the grayscale result in the 3D reconstruction result of the pixel location, and the displacement position corresponding to the maximum value of the first estimated curve is taken as the depth result in the 3D reconstruction result of the pixel location.

7. The three-dimensional reconstruction method as described in claim 1, characterized in that, After obtaining the 3D reconstruction result of the sample under test by performing 3D reconstruction based on the maximum pixel value at the same pixel position in all sharpness images, the 3D reconstruction method further includes: For each pixel position in the 3D reconstruction result of the sample to be tested, the pixels at the pixel positions whose sharpness values ​​do not meet the preset sharpness requirements are filtered out to obtain the pixels at the remaining pixel positions, and the 3D reconstruction result is regenerated based on the pixels at the remaining pixel positions.

8. A three-dimensional reconstruction device based on multidimensional modulation, applied in a microscopic reconstruction system, the microscopic reconstruction system comprising a projection unit, an imaging unit, and a motion unit, characterized in that, The three-dimensional reconstruction device includes: The unit control module is used to control the motion unit to drive the sample under test to move relative to the imaging unit along a preset direction; The pattern projection module is used to control the projection unit to project complementary lighting patterns and / or lighting patterns with different spatial scales onto the sample under test each time the module moves to a displacement position; wherein the complementary lighting patterns have different light intensity distribution characteristics at the same spatial position. An image acquisition module is used to control the imaging unit to acquire a projected image of the surface of the sample under test after each projection. The image processing module is used to generate a sharpness map based on the projected images acquired in each projection cycle; wherein, completing one projection of a lighting pattern with a complementary relationship or a lighting pattern with different spatial scales constitutes a projection cycle, and the sharpness map is used to characterize the sharpness of each pixel position. The 3D reconstruction module is used to perform 3D reconstruction based on the maximum pixel value at the same pixel position in all resolution images, so as to obtain the 3D reconstruction result of the sample under test.

9. A microscopic reconstruction system, characterized in that, include: A projection unit is used to project lighting patterns that have a complementary relationship and / or lighting patterns with different spatial scales; wherein the lighting patterns that have a complementary relationship have different light intensity distribution characteristics at the same spatial location; An imaging unit is used to acquire a projected image of the surface of the sample to be tested; A motion unit is used to drive the sample to be tested to move relative to the imaging unit along a preset direction, so that when it moves to a preset displacement position, the projection unit responds to the control command sent by the processor to project. A processor is used to process the sample under test using the three-dimensional reconstruction method based on multidimensional modulation according to any one of claims 1-7, and obtain the three-dimensional reconstruction result of the sample under test.

10. A computer program product comprising a computer program and / or instructions, characterized in that, When the computer program and / or instructions are executed by the processor, they implement the three-dimensional reconstruction method based on multidimensional modulation as described in any one of claims 1-7.

Images