Microscopic object three-dimensional reconstruction method based on scanning electron microscope imaging principle

Abstract

The invention discloses a microscopic object three-dimensional reconstruction method based on a scanning electron microscope imaging principle. The method comprises the following steps: firstly, based on a scanning electron microscope secondary electron imaging physical mechanism, establishing a mapping relation between an electron beam incident angle and an image gray scale, and reconstructing a Gaussian plane color model by calculating an included angle between a camera observation direction and a Gaussian plane normal direction and combining position related conductive characteristic parameters trained by a multi-layer perceptron; floating point artifacts generated by mismatch of physical principles are thoroughly eliminated; secondly, innovatively introducing a camera pose joint optimization mechanism, converting a camera space position and a rotation pose of an acquired image into trainable parameters, and synchronously optimizing a pose transformation matrix in a plane Gaussian sputtering frame training process, thereby effectively solving the problem of initial pose error amplification caused by a large depth-of-field characteristic of a scanning electron microscope, and improving the precision of a plane Gaussian sputtering frame. And the ghosting phenomenon of the reconstructed image is obviously inhibited. According to the method, high-fidelity microstructure reconstruction is finally realized, and an accurate three-dimensional representation means is provided for the field of material science.

Description

Technical Field

[0001] The present invention belongs to the intersection of electron microscopy and three-dimensional computer vision, and in particular relates to a three-dimensional reconstruction method for microscopic objects based on the imaging principle of a scanning electron microscope. Background Art

[0002] Accurate three-dimensional characterization of a material's microstructure is key to understanding its macroscopic properties. However, traditional microscopy techniques (such as scanning electron microscopy (SEM) and optical microscopy) mainly provide two-dimensional images that can only reflect cross-sectional morphology and cannot fully present three-dimensional spatial distribution characteristics. Since many material properties essentially depend on the three-dimensional structure, relying solely on two-dimensional characterization can easily lead to information loss or even misjudgment, making it difficult to meet the needs of modern material design and optimization. Therefore, it is crucial to develop characterization techniques that can obtain three-dimensional microstructures.

[0003] There are two main types of microscopic 3D reconstruction methods:

[0004] 1. Photometric Stereo: Using multiple light sources or detectors to capture images under varying illumination from a single viewing angle, the method analyzes brightness variations to infer surface normals and reconstructs the 3D topography. This method is highly efficient and real-time, suitable for samples with smooth, uniformly reflective surfaces. However, its limited viewing angle makes it difficult to capture the full view of complex structures, and its sample requirements make it difficult to reconstruct complex structures.

[0005] 2. Traditional stereo vision: A single detector captures multiple images from different viewpoints using a rotating sample stage. Feature matching algorithms analyze image differences, calculate spatial position, and reconstruct the 3D structure. This method offers a wide viewing angle and is suitable for samples with rough or textured surfaces. However, the resulting reconstruction is typically triangular facets, resulting in poor rendering quality, insufficient detail restoration, and sensitivity to errors and poor robustness.

[0006] More importantly, these two traditional methods do not fully consider the unique imaging principles of scanning electron microscope (SEM) images and lack targeted optimization, resulting in unsatisfactory results when processing SEM images.

[0007] At the same time, 3D reconstruction technology is undergoing a paradigm shift from traditional methods to neural rendering. Traditional stereo vision methods rely on sparse feature matching, resulting in reconstructions primarily of triangular facets with limited detail fidelity. Photometric stereo, while improving reconstruction efficiency, is limited by a single viewpoint and the sample's surface characteristics. With the rise of implicit representation methods such as Neural Radiance Field (NeRF), reconstruction quality has been significantly improved. However, these methods suffer from low computational efficiency and difficulty modeling high-frequency details.

[0008] In this context, 3D Gaussian Splatting (3DGS) technology enables real-time, high-fidelity rendering through explicit Gaussian distribution modeling, providing a new paradigm for microscopic 3D reconstruction. However, the standard 3DGS rendering pipeline is designed based on the physics of visible light imaging, and its color modeling relies on spherical harmonics to describe view-dependent reflections. This makes it unsuitable for the unique secondary electron imaging mechanism of scanning electron microscopes (SEMs), particularly the strong correlation between the electron beam incident angle and grayscale values.

[0009] To improve the ability to represent complex structures, researchers proposed Planar-based Gaussian Splatting for Efficient and High-Fidelity Surface Reconstruction (PGSR). This method compresses a three-dimensional Gaussian kernel into a two-dimensional Gaussian plane with normal constraints, enhancing the stability of the geometric representation. However, existing PGSR still has two major limitations:

[0010] 1. Imaging physics mismatch: Its spherical harmonic color model is difficult to fit the grayscale response in SEM images, which changes dramatically with the surface inclination angle, resulting in non-physical "floating point" artifacts in the reconstructed surface;

[0011] 2. Posture-sensitive defects: The large depth of field of SEM significantly amplifies the initial pose estimation error. The traditional PGSR lacks a posture optimization mechanism, which causes the "ghosting" phenomenon under multiple perspectives.

[0012] Therefore, developing a PGSR method for optimizing the physical properties of SEM imaging, establishing an explicit mathematical model of electron beam-sample interaction, and solving the problem of posture error transmission have become key breakthroughs in improving the accuracy of three-dimensional reconstruction of microscopic objects. Summary of the Invention

[0013] The present invention aims to solve the problem of low reconstruction quality of existing reconstruction methods on scanning electron microscope (SEM) images and proposes a three-dimensional reconstruction method for microscopic objects based on the SEM imaging principle.

[0014] The present invention improves the quality of SEM image reconstruction mainly through two approaches:

[0015] 1. Optimize the PGSR rendering pipeline: Establish a mathematical model based on the principle of SEM secondary electron imaging to make the rendering process closer to real SEM imaging:

[0016] ① Calculate the angle between the vector from the camera to the center of the Gaussian plane and the normal of the Gaussian plane, and establish its relationship with the color of the Gaussian plane;

[0017] ② Train a multi-layer perceptron (MLP) to fit the conductivity coefficients of different areas of the sample.

[0018] 2. Introducing a pose optimization module: Continuously optimize the camera pose during PGSR training to reduce pose errors and eliminate the "ghosting" problem in SEM image rendering.

[0019] The object of the present invention is achieved through the following technical solution: a method for three-dimensional reconstruction of microscopic objects based on the imaging principle of a scanning electron microscope, the method comprising the following steps:

[0020] Step 1: Design standard sample drawings and prepare samples using micro-nano printing technology;

[0021] Step 2: spraying a conductive coating on the sample surface and fixing it on the scanning electron microscope sample stage, tilting the sample stage, and collecting sample images at different angles;

[0022] Step 3: reconstruct the initial sparse 3D point cloud of the sample and the camera pose based on the image sequence;

[0023] Step 4: Optimize the PGSR rendering pipeline. Specifically, establish a mapping relationship between the electron beam incident angle and the image grayscale based on the scanning electron microscope imaging principle, calculate the angle between the ray vector from the camera to the center of the Gaussian plane and the normal vector of the Gaussian plane, and use a multi-layer perceptron to fit the position-related conductivity coefficient to correct the color representation of the Gaussian plane.

[0024] Step 5: Add a pose optimization module to PGSR. Specifically, define the pose parameters of each camera as trainable variables and continuously optimize its SE(3) transformation matrix through gradient descent during PGSR training.

[0025] Step 6: Use the optimized PGSR to train the initial point cloud data to achieve 3D reconstruction of the sample.

[0026] Furthermore, in the step 1, a sample drawing is designed using mechanical design software, and then a two-photon 3D printer is used for micro-nano printing.

[0027] Furthermore, in step 2, the sample stage is tilted from 0° to 35° in 5° steps along the negative direction of the x-axis, for a total of 8 tilt angles. At each tilt angle, the sample stage is rotated clockwise around its normal axis in 45° steps, for a total of 8 orientations. Images are collected at each combination of tilt angle and orientation, for a total of 64 times; the field of view and focal length need to be adjusted for each acquisition to ensure image clarity.

[0028] Furthermore, in step 4, the scanning electron microscope imaging principle is specifically as follows:

[0029] All SEM images are secondary electron images. The relationship between the SEM secondary electron emission coefficient and the electron beam incident angle is:

[0030] δ(θ)=δ(0)·(a(1-cosθ))

[0031] Where θ represents the incident angle of the electron beam, δ(θ) represents the secondary electron emission coefficient when the electron beam incident angle is θ, δ(0) represents the secondary electron emission coefficient when the electron beam is vertically incident, and a represents the conductivity of the sample.

[0032] The grayscale value of each pixel in the scanning electron microscope image ranges from 0 to 255. The grayscale value is positively correlated with the number of secondary electrons excited by the sample, and the number of secondary electrons is proportional to the emission coefficient. Therefore, the greater the inclination angle of the sample surface, the higher the image grayscale value; conversely, the grayscale value is lower.

[0033] Furthermore, in step 4, PGSR compresses the three-dimensional Gaussian kernel in space into a two-dimensional Gaussian plane. Each Gaussian plane contains the following characteristic parameters: spatial position p, covariance matrix ∑ describing the distribution characteristics, transparency parameter α, and color feature c represented by spherical harmonic function. Under the given camera pose, PGSR maps the Gaussian plane in space to the imaging plane through projection transformation. Subsequently, the color values ​​of N ordered Gaussian planes mapped to the same pixel position are superimposed and synthesized, and finally the composite color value of the pixel point is calculated. The calculation formula is:

[0034]

[0035] Among them, C represents the color of the pixel, c i Represents the color of each Gaussian plane at the pixel position, α i Indicates the transparency of each Gaussian plane at the pixel position; corrects the color of the Gaussian plane:

[0036] c i ′ =c i e μ(1-cos(θ))

[0037] Among them, c i ′ is the corrected color of each Gaussian plane; μ is the conductivity coefficient of each Gaussian plane, which is used to fit the conductivity coefficient of the sample; θ is the angle between the ray vector v from the camera to the center of the Gaussian plane and the normal vector n of the Gaussian plane, which is used to represent the incident angle of the electron beam;

[0038] The optimized PGSR rendering pixel color calculation formula is:

[0039]

[0040] At this point, the rendering pipeline optimization of PGSR is completed.

[0041] Furthermore, in step 4, the conductivity coefficient μ is output by multi-layer perceptron training. To improve the model's sensitivity to position changes, the input is the encoded Gaussian center position p = (x, y, z). The encoding method is defined as:

[0042] γ(p)=(sin(2 0 πp),cos(2 0 πp),…,sin(2 L-1 πp),cos(2 L-1 πp))

[0043] Where L is the encoding order; the encoded vector is input into a trainable multi-layer perceptron network, which contains two fully connected hidden layers. The output layer uses the Sigmoid activation function to constrain μ to the range [0,1]:

[0044] μ=σ(W2(W1x+b1)+b2)

[0045] Where x is the encoded vector, W1, W2, b1, b2 are trainable parameters, and σ(·) is the Sigmoid activation function.

[0046] Furthermore, in step 4, the cosine value cosθ of the angle between the ray vector v from the camera to the center of the Gaussian plane and the normal vector n of the Gaussian plane is calculated as follows:

[0047]

[0048] Among them, PGSR compresses the Gaussian ellipsoid along its shortest axis into a Gaussian plane, and the direction of its shortest axis is the Gaussian plane normal vector n.

[0049] Furthermore, in step 5, all camera poses are set to the six parameters to be optimized (x, y, z, pitch, yaw, roll), and then (pitch, yaw, roll) is converted into quaternion q = (q w ,q x ,q y ,q z ), and then converted into SE(3) format to ensure the differentiability of the six parameters of the pose; assuming that the rotation order of the Euler angle is ZYX, the quaternion calculation formula is:

[0050]

[0051] q w =c1c2c3+s1s2s3

[0052] q x =s1s2c s +c1c2s3

[0053] q y =c1s2c3-s1c2s3

[0054] q z =c1c2s3-s1s2c3

[0055] Quaternion q=(q w ,q x ,q y ,q z )The corresponding rotation matrix R∈SO(3) is:

[0056]

[0057] The translation t = (x, y, z) and the rotation matrix R are combined into the transformation matrix T∈SE(3):

[0058] t=[x,y,z] T

[0059] After assigning each camera its corresponding transformation matrix T, during the training process, the camera pose parameters are optimized by calculating the loss function and back-propagating the gradient.

[0060] Furthermore, the loss function used in step 6 is the RGB difference L between the PGSR rendered image and the real image of the training set. rgb , the formula is:

[0061] L rgb =I render -I train

[0062] Among them, I render RGB value of the PGSR rendered image, I train is the RGB value of the real image in the training set; during the training process, the gradient descent algorithm is used to continuously reduce I render And optimize the Gaussian distribution to finally complete the three-dimensional reconstruction of the sample.

[0063] The beneficial effects of the present invention are as follows:

[0064] 1. A new rendering pipeline is proposed. By calculating the angle between the ray vector from the camera to the center of the Gaussian plane and the normal vector of the Gaussian plane, fitting the conductivity coefficient of different areas of the sample, and correcting the color of the Gaussian plane, it effectively solves the "floating point" artifact problem in the three-dimensional reconstruction results of scanning electron microscope images.

[0065] 2. By continuously optimizing the posture of the SEM camera during PGSR training, the posture error is significantly reduced, effectively solving the "ghosting" problem in the 3D reconstruction results of the SEM image. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A flowchart of a method for three-dimensional reconstruction of microscopic objects based on the imaging principle of a scanning electron microscope provided in an embodiment of the present invention;

[0067] Figure 2 A standard sample drawing provided for an embodiment of the present invention;

[0068] Figure 3 A graph showing the relationship between the secondary electron emission coefficient of a scanning electron microscope and the electron beam incident angle provided by an embodiment of the present invention;

[0069] Figure 4 A diagram showing the relationship between the Gaussian plane and the scanning electron microscope camera provided in an embodiment of the present invention;

[0070] Figure 5 A schematic diagram of an optimized rendering pipeline based on scanning electron microscope imaging principles provided by an embodiment of the present invention;

[0071] Figure 6 A schematic diagram of a method for solving the "floating point" problem of reconstructed samples provided by an embodiment of the present invention;

[0072] Figure 7 Schematic diagram of the method for solving the "ghosting" problem of reconstructed samples provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0073] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0074] like Figure 1 As shown, the method for three-dimensional reconstruction of microscopic objects based on the scanning electron microscope imaging principle provided by the embodiment of the present invention specifically includes the following steps:

[0075] 1. Design standard sample drawings and prepare samples using micro-nano printing technology.

[0076] like Figure 2 As shown in the figure, a standard sample drawing was designed using the mechanical design software SolidWorks. Its overall dimensions are approximately 200 microns, with a rectangular base and a frustum. Subsequently, a two-photon 3D printer was used to produce the sample solid body from resin material with a machining accuracy of 0.5 microns.

[0077] 2. Spray a conductive coating on the sample surface and attach the sample base to the SEM sample stage; tilt the sample stage to obtain sample images at different angles.

[0078] The sample surface was gold-sprayed for one minute and then affixed to the center of the SEM stage. The stage was then tilted along the negative x-axis in 5° increments from 0° to 35° (eight tilt angles total). At each tilt angle, the stage was rotated clockwise around its normal axis in 45° increments (0° to 360°, eight positions total). A total of 64 images were acquired at each combination of these tilt and orientations. The field of view and focus were adjusted for each acquisition to ensure image clarity.

[0079] 3. Calculate the initial sparse point cloud and camera pose of the sample.

[0080] All captured images were input into Metashape, a real-world 3D modeling software. The point cloud accuracy was set to "very high." The software then ran the program and exported the initial sparse point cloud of the sample and the initial camera pose corresponding to each image. This initial sparse point cloud and camera pose will serve as the basic training data for PGSR.

[0081] 4. Based on the principle of scanning electron microscope imaging, optimize the rendering pipeline of PGSR.

[0082] The scanning electron microscope images used in the present invention are all secondary electron images, and the imaging process can be summarized as follows: in a vacuum environment, the electron gun in the scanning electron microscope cavity emits a high-energy electron beam; after being focused and deflected by a series of electromagnetic lenses, the electron beam bombards a specific point on the sample surface and stimulates secondary electrons; a dedicated probe collects these secondary electrons, converts the electrical signals generated by them into light signals, and assigns grayscale values ​​to the corresponding pixels in the image; finally, the scanning electron microscope generates a grayscale image of the sample through a "line-by-line scanning, point-by-point imaging" method.

[0083] The number of secondary electrons excited by a high-energy electron beam is related to its incident angle, such as Figure 3 As shown in Figure 2, the relationship between the secondary electron emission coefficient of the scanning electron microscope and the electron beam incident angle is:

[0084] δ(θ)=δ(0)·(a(1-cosθ))

[0085] Where θ represents the incident angle of the electron beam (relative to the normal line of the sample surface), δ(θ) represents the secondary electron emission coefficient when the electron beam incident angle is θ, and δ(0) represents the secondary electron emission coefficient when the electron beam is vertically incident, a represents the conductivity of the sample, Figure 3 L represents the depth of the electron beam bombarding the sample.

[0086] The grayscale value of each pixel in a SEM image ranges from 0 to 255. This grayscale value is positively correlated with the number of secondary electrons emitted by the sample, which in turn is proportional to the emission coefficient. Therefore, locations with greater surface inclination angles have higher image grayscale values ​​(brighter); conversely, locations with lower surface inclination angles have lower grayscale values ​​(darker). This results in significant differences in image grayscale values ​​when observing the same location on the sample from different angles. This phenomenon is the root cause of "floating point" artifacts when PGSR is trained on SEM images.

[0087] PGSR flattens the three-dimensional Gaussian kernel in space into a two-dimensional Gaussian plane, and introduces the normal vector of the Gaussian plane to perform relevant geometric constraints and optimize the imaging quality. In PGSR, each Gaussian plane contains the following characteristic parameters: spatial position p, covariance matrix ∑ describing the distribution characteristics, transparency parameter α, and color feature c represented by spherical harmonics (SH). Under given camera pose conditions, PGSR maps the Gaussian plane in space to the imaging plane through projection transformation. Subsequently, PGSR uses the α-blending algorithm to superimpose and synthesize the color values ​​of N ordered Gaussian planes mapped to the same pixel position, and finally calculates the composite color value of the pixel. The color calculation process can be expressed by the following mathematical expression:

[0088]

[0089] Among them, C represents the color of the pixel, c i Represents the color of each Gaussian plane at the pixel position, α i Indicates the transparency of each Gaussian plane at the pixel location.

[0090] In SEM image training, spherical harmonics can only partially fit the color changes related to viewing angles, resulting in "floating point" artifacts in the reconstructed scene. Therefore, the color of the Gaussian plane is corrected:

[0091] c i ′ =c i e μ(1-cos(θ))

[0092] Among them, c i ′ is the corrected color of each Gaussian plane; μ is the conductivity coefficient of each Gaussian plane, which is used to fit the conductivity coefficient of the sample; θ is the angle between the ray vector v from the camera to the center of the Gaussian plane and the normal vector n of the Gaussian plane, which is used to represent the incident angle of the electron beam, such as Figure 4 shown.

[0093] Specifically, μ is calculated by a multi-layer perceptron (MLP), and the specific process is as follows:

[0094] For each Gaussian plane center position p = (x, y, z), feature enhancement is performed through 3-order position encoding:

[0095] γ(p)=(sin(2 0 πp),cos(2 0 πp),sin(2 1 πp),cos(2 1 πp)),sin(2 2 πp),cos(2 2 πp))

[0096] Here, γ(p) represents the enhanced feature; an 18-dimensional embedding vector is generated for each position, and the encoded vector is input into a trainable MLP network, which consists of two fully connected hidden layers (64 neurons each). The output layer uses a Sigmoid activation function to constrain μ to the range [0,1]:

[0097]

[0098] Among them, x is the encoded 18-dimensional position vector, W1, W2, b1, b2 are trainable parameters, and σ(z) is the Sigmoid activation function.

[0099] The formula for calculating the cosine value cosθ of the angle between the ray vector v from the camera to the center of the Gaussian plane and the Gaussian plane normal vector n is as follows:

[0100]

[0101] Among them, PGSR compresses the Gaussian ellipsoid along its shortest axis into a Gaussian plane, and the direction of its shortest axis is the Gaussian plane normal vector n.

[0102] Finally, the optimized PGSR rendering pixel color calculation formula is:

[0103]

[0104] At this point, the rendering pipeline optimization of PGSR is completed. The flowchart of the optimization part is as follows Figure 5 shown. Figure 6 The figure shows the "floating point" problem of SEM images in PGSR, as well as the rendering effect after pipeline optimization. From left to right, they are the true value image, the PGSR rendered image, and the image rendered by the method of the present invention.

[0105] 5. Add pose optimization module to PGSR.

[0106] Due to the large depth of field of SEM images, there are significant errors in the initial pose estimation based on traditional feature matching. Such errors are manifested as multi-view consistency degradation in the PGSR framework, specifically reflected as "ghosting" artifacts in the rendered image (such as Figure 7 The fundamental reason is that the pose deviation causes the photometric consistency constraint of the training set images to fail, making it impossible for the Gaussian plane to fit the scene geometry to converge to the optimal solution.

[0107] Therefore, a pose optimization module is added to PGSR:

[0108] Set all camera poses to six parameters that can be optimized (x, y, z, pitch, yaw, roll), and then convert (pitch, yaw, roll) into quaternion q = (q w ,q x ,q y ,q z ), and then converted to SE(3) format to ensure the differentiability of the six parameters of the pose. In this example, assuming that the rotation order of the Euler angle is ZYX, the quaternion calculation formula is:

[0109]

[0110] q w =c1c2c3+s1s2s3

[0111] q x =s1s2c s +c1c2s3

[0112] q y =c1s2c3-s1c2s3

[0113] q z =c1c2s3-s1s2c3

[0114] Quaternion q=(q w ,q x ,q y ,q z )The corresponding rotation matrix R∈SO(3) is:

[0115]

[0116] The translation t = (x, y, z) and the rotation matrix R are combined into the transformation matrix T∈SE(3):

[0117] t=[x,y,z] T

[0118] After assigning each camera its corresponding transformation matrix T, during the training process, the camera pose parameters are optimized by calculating the loss function and back-propagating the gradient.

[0119] Figure 7 The "ghosting" problem of SEM images in PGSR is demonstrated, as well as the rendering effect after adding the pose optimization module. From left to right are the true value image, the PGSR rendered image, and the image rendered by the method of the present invention.

[0120] 6. Use the optimized PGSR to train the initial point cloud data and complete the 3D reconstruction of the sample.

[0121] The loss function used in this example is the RGB difference L between the PGSR rendered image and the real image in the training set. rgb , the formula is:

[0122] L rgb =I render -I train

[0123] Among them, I render RGB value of the PGSR rendered image, I train is the RGB value of the real image in the training set. During the training process, the gradient descent algorithm is used to continuously reduce I render And optimize the Gaussian distribution to finally complete the three-dimensional reconstruction of the sample.

[0124] The embodiment of the present invention verifies the effectiveness of the proposed method through specific experiments. A standard sample of about 200 microns (including a base and a frustum structure) was prepared using a two-photon printer. After gold spraying, a total of 64 high-definition secondary electron images were collected in a scanning electron microscope (SEM) according to a preset combination of inclination angles (0° to 35°, step length 5°) and azimuth angles (0° to 360°, step length 45°). The image sequence was processed using MetaShape software to generate a high-precision initial sparse point cloud and camera pose estimation. Based on the principle of SEM secondary electron imaging (the physical relationship between the emission coefficient δ(θ) and the incident angle θ), the PGSR rendering pipeline was key optimized: by calculating the angle θ between the camera ray vector v and the Gaussian plane normal vector n, and using a multi-layer perceptron (MLP) to fit the position-related conductivity coefficient μ, the color representation of the Gaussian plane was corrected, and the "floating point" artifacts in the reconstruction results (such as Figure 6 At the same time, a differentiable camera pose optimization module is introduced to define the pose parameters of each camera as trainable variables, and its SE(3) transformation matrix is ​​continuously optimized by gradient descent during the PGSR training process, which effectively solves the problem of initial pose estimation error amplification caused by the large depth of field of SEM and significantly reduces the "ghosting" phenomenon (such as Figure 7Finally, the optimized PGSR framework was used to train the initial point cloud data, and the RGB difference between the rendered image and the real image was used as the loss function for optimization. This successfully achieved high-fidelity, artifact-free 3D reconstruction of a standard sample. This example fully demonstrates the significant effectiveness of the rendering pipeline optimization and pose optimization module in resolving artifacts unique to SEM image reconstruction and improving the accuracy of 3D reconstruction of microscopic objects.

[0125] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the concept and scope of the present invention. Any modifications and improvements made to the technical solution of the present invention by a person of ordinary skill in the art without departing from the design concept of the present invention shall fall within the scope of protection of the present invention. The technical content for which protection is sought in the present invention is fully set forth in the claims.

Claims

1. A method for three-dimensional reconstruction of microscopic objects based on the imaging principle of scanning electron microscopy, characterized in that: include: Step 1: Design standard sample drawings and prepare samples using micro-nano printing technology; Step 2: spraying a conductive coating on the sample surface and fixing it on the scanning electron microscope sample stage, tilting the sample stage, and collecting sample images at different angles; Step 3: reconstruct the initial sparse 3D point cloud of the sample and the camera pose based on the image sequence; Step 4: Optimize the PGSR rendering pipeline. Specifically, establish a mapping relationship between the electron beam incident angle and the image grayscale based on the scanning electron microscope imaging principle, calculate the angle between the ray vector from the camera to the center of the Gaussian plane and the normal vector of the Gaussian plane, and use a multi-layer perceptron to fit the position-related conductivity coefficient to correct the color representation of the Gaussian plane. Step 5: Add a pose optimization module to PGSR. Specifically, define the pose parameters of each camera as trainable variables and continuously optimize its SE(3) transformation matrix through gradient descent during PGSR training. Step 6: Use the optimized PGSR to train the initial point cloud data to achieve 3D reconstruction of the sample.

2. The method for three-dimensional reconstruction of microscopic objects based on the scanning electron microscope imaging principle according to claim 1, characterized in that: In the step 1, a sample drawing is designed using mechanical design software, and then a two-photon 3D printer is used for micro-nano printing.

3. The method for three-dimensional reconstruction of microscopic objects based on the scanning electron microscope imaging principle according to claim 1, characterized in that: In step 2, the sample stage is tilted from 0° to 35° in 5° steps along the negative direction of the x-axis, for a total of 8 tilt angles. At each tilt angle, the sample stage is rotated clockwise around its normal axis in 45° steps, for a total of 8 orientations. Images are collected at each combination of tilt angle and orientation, for a total of 64 times; the field of view and focal length need to be adjusted for each acquisition to ensure image clarity.

4. The method for three-dimensional reconstruction of microscopic objects based on the scanning electron microscope imaging principle according to claim 1, characterized in that: In step 4, the scanning electron microscope imaging principle is specifically as follows: All SEM images are secondary electron images. The relationship between the SEM secondary electron emission coefficient and the electron beam incident angle is δ(θ) = δ(0)·(α(1-cosθ)) Where θ represents the incident angle of the electron beam, δ(θ) represents the secondary electron emission coefficient when the electron beam incident angle is θ, δ(0) represents the secondary electron emission coefficient when the electron beam is vertically incident, and a represents the conductivity of the sample. The grayscale value of each pixel in the scanning electron microscope image ranges from 0 to 255. The grayscale value is positively correlated with the number of secondary electrons excited by the sample, and the number of secondary electrons is proportional to the emission coefficient. Therefore, the greater the inclination angle of the sample surface, the higher the image grayscale value; conversely, the grayscale value is lower.

5. The method for three-dimensional reconstruction of microscopic objects based on the scanning electron microscope imaging principle according to claim 1, characterized in that: In step 4, PGSR compresses the three-dimensional Gaussian kernel in space into a two-dimensional Gaussian plane. Each Gaussian plane contains the following characteristic parameters: spatial position p, covariance matrix ∑ describing distribution characteristics, transparency parameter α, and color feature c represented by spherical harmonic function. Under given camera pose conditions, PGSR maps the Gaussian plane in space to the imaging plane through projection transformation. Subsequently, the color values ​​of N ordered Gaussian planes mapped to the same pixel position are superimposed and synthesized, and finally the composite color value of the pixel point is calculated. The calculation formula is: Among them, C represents the color of the pixel, c i Represents the color of each Gaussian plane at the pixel position, α i Indicates the transparency of each Gaussian plane at the pixel position; corrects the color of the Gaussian plane: c′ i =c i e μ(1-cos(θ)) Among them, c′ i is the corrected color of each Gaussian plane; μ is the conductivity coefficient of each Gaussian plane, which is used to fit the conductivity coefficient of the sample; θ is the angle between the ray vector v from the camera to the center of the Gaussian plane and the normal vector n of the Gaussian plane, which is used to represent the incident angle of the electron beam; The optimized PGSR rendering pixel color calculation formula is: At this point, the rendering pipeline optimization of PGSR is completed.

6. The method for three-dimensional reconstruction of microscopic objects based on the scanning electron microscope imaging principle according to claim 5, characterized in that: In step 4, the conductivity coefficient μ is output by multi-layer perceptron training. To improve the model's sensitivity to position changes, the input is the encoded Gaussian center position p = (x, y, z). The encoding method is defined as: γ(p)=(sin(2 0 πp),cos(2 0 πp),…,sin(2 L-1 πp),cos(2 L-1 πp)) Where L is the encoding order; the encoded vector is input into a trainable multi-layer perceptron network, which contains two fully connected hidden layers. The output layer uses the Sigmoid activation function to constrain μ to the range [0,1]: σ=μ(W2(W1x+b1)+b2) Where x is the encoded vector, W1, W2, b1, b2 are trainable parameters, and σ(·) is the Sigmoid activation function.

7. The method for three-dimensional reconstruction of microscopic objects based on the scanning electron microscope imaging principle according to claim 5, characterized in that: In step 4, the cosine value cosθ of the angle between the ray vector v from the camera to the center of the Gaussian plane and the normal vector n of the Gaussian plane is calculated as follows: Among them, PGSR compresses the Gaussian ellipsoid along its shortest axis into a Gaussian plane, and the direction of its shortest axis is the Gaussian plane normal vector n.

8. The method for three-dimensional reconstruction of microscopic objects based on the scanning electron microscope imaging principle according to claim 1, characterized in that: In step 5, all camera poses are set to the six parameters to be optimized (x, y, z, pitch, yaw, roll), and then (pitch, yaw, roll) is converted into quaternion q = (q w ,q x ,q y ,q z ), and then converted into SE(3) format to ensure the differentiability of the six parameters of the pose; assuming that the rotation order of the Euler angle is ZYX, the quaternion calculation formula is: q w =c1c2c3+s1s2s3 q x =s1s2c s +c1c2s3 q y =c1s2c3-s1c2s3 q z =c1c2s3-s1s2c3 Quaternion q=(q w ,q x ,q y ,q z )The corresponding rotation matrix R∈SO(3) is: The translation t = (x, y, z) and the rotation matrix R are combined into the transformation matrix T∈SE(3): t=[x,y,z] T After assigning each camera its corresponding transformation matrix T, during the training process, the camera pose parameters are optimized by calculating the loss function and back-propagating the gradient.

9. The method for three-dimensional reconstruction of microscopic objects based on the scanning electron microscope imaging principle according to claim 1, characterized in that: The loss function used in step 6 is the RGB difference L between the PGSR rendered image and the real image of the training set. rgb , the formula is: L rgb =I render -I train Among them, I render RGB value of the PGSR rendered image, I train is the RGB value of the real image in the training set; during the training process, the gradient descent algorithm is used to continuously reduce I render And optimize the Gaussian distribution to finally complete the three-dimensional reconstruction of the sample.

Images