Processing method and device for scanning electron microscope graph simulation
By using 3D voxel files or generative geometric parameter modeling, combined with Marching Cubes algorithm and multiphysics simulation, the gap between simulation results and intuitive morphological cognition in scanning electron microscopy simulation is solved, realizing end-to-end automated processing and information-enhanced image fusion.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING DP TECH CO LTD
- Filing Date
- 2025-12-12
- Publication Date
- 2026-04-17
AI Technical Summary
Existing scanning electron microscope simulation technology suffers from a gap between simulation results and intuitive morphological cognition, lacks end-to-end automated processing and information fusion capabilities, and struggles to generate information-enhanced fused images.
By modeling with 3D voxel files or generative geometric parameters, and extracting surface mesh networks using the Marching Cubes algorithm, the pose of the sample models is adjusted and multiphysics simulation is performed to generate rigorously registered visual and electron micrographs, achieving pixel-level feature fusion.
It bridges the cognitive gap between intuitive shapes and physical signals, constructs an end-to-end automated processing flow, generates information-enhanced fused images, and supports high-throughput simulation.
Smart Images
Figure CN121883758A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, and in particular to a processing method and apparatus for simulating scanning electron microscope images. Background Technology
[0002] Scanning electron microscopy (SEM) is a crucial tool for micro- and nano-scale morphology and composition analysis in materials science, life sciences, and nanotechnology. Its core principle involves scanning a sample with a focused electron beam, generating high-resolution images by detecting the resulting backscattered electron (BSE) and secondary electron (SE) signals. BSE images are sensitive to atomic numbers and are commonly used for composition analysis; SE images are sensitive to surface morphology and are used to reveal fine structures. However, real-world SEM experiments face numerous challenges: high-end equipment is expensive, experimental cycles are long, parameter optimization relies heavily on experience, and there are irreversible risks associated with precious or easily damaged samples.
[0003] To overcome these limitations, computational simulation technology has emerged, using the Monte Carlo method to simulate the interaction between electrons and matter, providing a powerful tool for predicting experimental results, optimizing parameters, and interpreting images. However, existing SEM simulation schemes have the following shortcomings: 1) Existing schemes typically perform physical simulations in isolation, lacking coordination with high-fidelity visual rendering, resulting in a gap between the simulation results (grayscale electron microscope images) and researchers' intuitive understanding of 3D morphology, making it difficult to quickly establish the correspondence between signals and structures; 2) The failure to achieve end-to-end automated processing from geometric modeling and attitude control to multiphysics simulation greatly limits the efficiency of high-throughput simulation; 3) Existing technologies lack a unified framework to fuse physical signals (BSE / SE) and visual features (normals, depth), making it impossible to generate information-enhanced fused images to uncover deeper sample information. Summary of the Invention
[0004] The purpose of this invention is to address the shortcomings of existing technologies by providing a processing method, apparatus, electronic device, and computer-readable storage medium for simulating scanning electron microscope (SEM) images. This invention can directly obtain a sample model by modeling the sample voxel structure from a 3D voxel file, or it can first perform 3D modeling and voxel file extraction based on generative geometric parameters, and then obtain a sample model based on the voxel file. After obtaining the sample model, a surface mesh network is obtained by extracting the triangular mesh network of the sample surface using the Marching Cubes algorithm. Then, the sample model and the surface mesh network are synchronously translated and rotated, and after pose adjustment, a closed environment space (a six-sided cuboid structure) enclosing the sample model is modeled. Then, light source points are set on the positive Z-axis above the sample model, and camera points are set on the positive Z-axis above the model. Finally, the normals, depths, and RGB features of each pixel on the camera imaging plane are simulated according to the orthogonal camera imaging principle to obtain a first visual image. Then, the electron gun is set on the positive Z-axis above the sample model; the electron beam scanning plane of the electron gun is set to a spatial plane with the same spatial position, size, and grid resolution as the camera imaging plane, ensuring that the scanning points of the electron beam correspond one-to-one with the pixels of the visual image; the corresponding BSE and SE detector signal receiving ranges are set on the electron beam scanning plane; and the BSE / SE signal characteristics of each scanning point on the electron beam scanning plane are simulated according to the working principle of scanning electron microscope to obtain the corresponding first / second electron microscope images. Then, pixel-level feature fusion is performed on the first / second electron microscope images and the first visual image respectively to obtain the corresponding first / second electron microscope image fusion image. This invention can generate strictly registered visual images and electron microscope images, bridging the cognitive gap between intuitive morphology and physical signals; this invention constructs an end-to-end fully automated pipeline process from geometric modeling to attitude control to multiphysics simulation, solving the problem of process fragmentation and providing core technical support for high-throughput simulation; this invention provides a pixel-level fusion framework for physical signals and visual features, generating an information-enhanced fused image by fusing BSE / SE signals with geometric features such as normals and depth.
[0005] To achieve the above objectives, a first aspect of the present invention provides a processing method for simulating scanning electron microscope images, the method comprising: The system receives simulation task data input by the user; the simulation task data includes a sample generation mode and modeling reference data; the sample generation mode includes a first mode and a second mode; when the sample generation mode is the first mode, the modeling reference data is a three-dimensional voxel file; when the sample generation mode is the second mode, the modeling reference data is a set of generative geometric parameters. Based on the simulated task data, sample voxel structure modeling is performed to obtain the corresponding sample model; and based on the Marching Cubes algorithm, triangular mesh network is extracted from the sample surface of the sample model to obtain the corresponding surface mesh network; The sample model is translated so that its base surface is on the XOY plane of the XYZ coordinate system, and the center of the projection of the sample model on the XOY plane is the origin O of the coordinate system. The translated sample model is then rotated according to preset X-axis and Y-axis rotation angles. After the rotation, a closed environment space enclosing the sample model is modeled. During the translation and rotation of the sample model, the surface mesh network is simultaneously translated and rotated. The spatial structure of the closed environment space is a hexagonal cuboid. A light source point is set on the positive Z-axis above the sample model, and the light intensity of the light source point is set. A camera point is also set on the positive Z-axis above the sample model. A camera imaging plane is set between the camera point and the sample model. The normal, depth, and RGB features of each pixel on the camera imaging plane are simulated according to the orthogonal camera imaging principle to obtain the corresponding first visual image. The camera imaging plane is parallel to the XOY plane. An electron gun is set on the positive Z-axis above the sample model; the electron beam scanning plane of the electron gun is set to a spatial plane with the same spatial position, size, and grid resolution as the camera imaging plane, ensuring that the scanning points of the electron beam correspond one-to-one with the pixels of the visual image; the corresponding BSE and SE detector signal receiving ranges are set on the electron beam scanning plane; and the signal characteristics of backscattered electrons and secondary electrons at each scanning point on the electron beam scanning plane are simulated according to the working principle of scanning electron microscope to obtain the corresponding first electron microscope image and second electron microscope image; the scanning points on the electron beam scanning plane correspond one-to-one with the pixels of the camera imaging plane. The first and second electron microscope images are respectively fused with the first visual image at the pixel level to obtain the corresponding first fused image and second fused image; and the first simulation report, composed of the sample model, the surface mesh network, the first visual image, the first electron microscope image, the second electron microscope image, the first fused image and the second fused image, is fed back to the current user.
[0006] Preferably, the step of obtaining the corresponding sample model by modeling the sample voxel structure based on the simulated task data specifically includes: Step 21: Identify the sample generation mode; if the sample generation mode is the first mode, then use the three-dimensional voxel file of the modeling reference data as the corresponding voxel set V; if the sample generation mode is the second mode, then perform three-dimensional structure modeling based on the generative geometric parameters of the modeling reference data, and extract voxel data from the current three-dimensional structure and form the corresponding voxel set V from the extracted data. The generative geometric parameters include at least fiber structure geometric parameters and carbon fiber structure geometric parameters; the voxel set V includes multiple voxel points v; each voxel point v includes voxel coordinates and voxel feature values. Step 22: Binarize all the voxel feature values of the voxel set V, reset the voxel feature values that are greater than or equal to a preset voxel feature threshold to 1, and reset the voxel feature values that are less than the voxel feature threshold to 0; Step 23: Construct the voxel mesh space corresponding to the voxel set V in the XYZ coordinate system and denote it as the current mesh space; and in the current mesh space, first denoise the voxel set V based on the morphological opening operation principle, and then fill the holes in the denoised voxel set V based on the morphological closing operation principle; The shape of the voxel grid space is a cuboid. Step 24: Based on the preset sample 3D size, randomly cut out a subspace from the current grid space as the corresponding sample grid space; and form the sample model by all the voxel points v corresponding to the sample grid space. The three-dimensional dimensions of the sample grid space satisfy the three-dimensional dimensions of the sample.
[0007] Preferably, the step of simulating the normals, depths, and RGB features of each pixel on the camera imaging plane according to the orthogonal camera imaging principle to obtain the corresponding first visual image specifically includes: Step 31: According to the parallel projection principle of orthogonal camera imaging, the corresponding surface points of each pixel on the camera imaging plane on the surface grid network are identified and recorded as the corresponding first surface points. Step 32: Calculate the corresponding first normal direction for each first surface point on the surface mesh network; Step 33: Calculate the direction of the light rays from the point light source to each of the first surface points to obtain the corresponding first light ray direction; Step 34: According to the Lambertian diffuse reflection model, calculate the corresponding first RGB pixel value of the current surface point based on the light source intensity and the first normal direction and the first light direction of each first surface point. Step 35: Calculate the vertical distance from each of the first surface points to the camera imaging plane and use the calculation result as the corresponding first depth; Step 36: The first pixel feature is composed of the first normal direction, the first depth and the first RGB pixel value corresponding to each first surface point; Step 37: The first visual image is composed of the first pixel features of all pixels on the camera imaging plane.
[0008] Preferably, the step of simulating the signal characteristics of backscattered electrons and secondary electrons at each scanning point on the electron beam scanning plane according to the working principle of the scanning electron microscope to obtain the corresponding first electron microscope image and second electron microscope image specifically includes: Step 41: Set the energy of the electron beam emitted by the electron gun to each scanning point; The electron gun scans each scanning point on the electron beam scanning plane point by point; and emits a corresponding electron beam to the current scanning point during each scanning, with the incident direction of the electron beam perpendicular to the electron beam scanning plane; the electron beam corresponds one-to-one with the scanning point. Step 42: Using the Monte Carlo simulation method, simulate the trajectory of each electron in each electron beam after entering the sample model; Step 43: Based on the electron simulation trajectory, the number of electrons that reach the signal receiving range of the BSE detector in each electron beam is counted to obtain the corresponding BSE signal strength. Step 44: Based on the electron simulation trajectory, the number of electrons that reach the SE detector signal receiving range in each electron beam is counted to obtain the corresponding SE signal intensity. Step 45: Perform grayscale conversion on the BSE signal intensity corresponding to each scanning point to obtain the corresponding first pixel grayscale; and combine all the obtained first pixel grayscales to form the corresponding first electron microscope image; Step 46: Perform grayscale conversion on the SE signal intensity corresponding to each scanning point to obtain the corresponding second pixel grayscale; and use all the obtained second pixel grayscales to form the corresponding second electron microscope image.
[0009] A second aspect of the present invention provides an apparatus for implementing the processing method for simulating scanning electron microscope images as described in the first aspect above. The apparatus includes: a data receiving module, a sample voxel and sample surface modeling module, a sample pose adjustment and sample environment modeling module, a visual imaging rendering module, a scanning electron microscope image simulation module, and a multimodal fusion module. The data receiving module is used to receive simulation task data input by the user; the simulation task data includes sample generation mode and modeling reference data; the sample generation mode includes a first mode and a second mode; when the sample generation mode is the first mode, the modeling reference data is a three-dimensional voxel file; when the sample generation mode is the second mode, the modeling reference data is a set of generative geometric parameters. The sample voxel and sample surface modeling module performs sample voxel structure modeling based on the simulation task data to obtain the corresponding sample model; and extracts the corresponding surface mesh network by performing triangular mesh network extraction on the sample surface of the sample model based on the Marching Cubes algorithm. The sample pose adjustment and sample environment modeling module is used to translate the sample model so that its base surface is on the XOY plane of the XYZ coordinate system, and the projection center of the sample model on the XOY plane is the origin O of the coordinate system; and to rotate the translated sample model according to preset X-axis and Y-axis rotation angles; and to model the closed environment space that can surround the sample model after the rotation; wherein, when translating and rotating the sample model, the surface mesh network is translated and rotated synchronously; the spatial structure of the closed environment space is a hexagonal cuboid structure; The visual imaging rendering module is used to set light source points on the positive Z-axis above the sample model, set the light intensity of the light source points, and set camera points on the positive Z-axis above the sample model; and set a camera imaging plane between the camera points and the sample model; and simulate the normals, depths, and RGB features of each pixel on the camera imaging plane according to the orthogonal camera imaging principle to obtain the corresponding first visual image; wherein, the camera imaging plane is parallel to the XOY plane; The scanning electron microscope (SEM) image simulation module is used to set the electron gun on the positive Z-axis above the sample model; and to set the electron beam scanning plane of the electron gun to a spatial plane with the same spatial position, size, and grid resolution as the camera imaging plane, ensuring that the scanning points of the electron beam correspond one-to-one with the pixels of the visual image; and to set the corresponding BSE and SE detector signal receiving ranges on the electron beam scanning plane; and to simulate the signal characteristics of backscattered electrons and secondary electrons at each scanning point on the electron beam scanning plane according to the working principle of the SEM to obtain the corresponding first and second SEM images; the scanning points on the electron beam scanning plane correspond one-to-one with the pixels of the camera imaging plane. The multimodal fusion module is used to perform pixel-level feature fusion of the first and second electron microscope images with the first visual image to obtain corresponding first fused image and second fused image; and the first simulation report composed of the sample model, the surface mesh network, the first visual image, the first electron microscope image, the second electron microscope image, the first fused image and the second fused image is fed back to the current user.
[0010] A third aspect of the present invention provides an electronic device, including: a memory, a processor, and a transceiver; The processor is used to couple with the memory, read and execute instructions in the memory to implement the steps of the method described in the first aspect above; The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.
[0011] A fourth aspect of the present invention provides a computer-readable storage medium storing computer instructions that, when executed by a computer, cause the computer to perform the instructions described in the first aspect.
[0012] This invention provides a processing method, apparatus, electronic device, and computer-readable storage medium for simulating scanning electron microscope (SEM) images. As described above, this invention can directly obtain a sample model by modeling the sample voxel structure from a 3D voxel file, or it can first perform 3D modeling and voxel file extraction based on generative geometric parameters, and then obtain a sample model based on the voxel file. After obtaining the sample model, a surface mesh network is obtained by extracting the triangular mesh network of the sample surface using the Marching Cubes algorithm. Then, the sample model and the surface mesh network are synchronously translated and rotated, and after pose adjustment, a closed environment space (a six-sided cuboid structure) that surrounds the sample model is modeled. Then, light source points are set on the positive Z-axis above the sample model, and camera points are set on the positive Z-axis above the model. Finally, the normals, depths, and RGB features of each pixel on the camera imaging plane are simulated according to the orthogonal camera imaging principle to obtain a first visual image. Then, the electron gun is set on the positive Z-axis above the sample model; the electron beam scanning plane of the electron gun is set to a spatial plane with the same spatial position, size, and grid resolution as the camera imaging plane, ensuring a one-to-one correspondence between the scanning points of the electron beam and the pixels of the visual image; the corresponding BSE and SE detector signal receiving ranges are set on the electron beam scanning plane; and the BSE / SE signal characteristics of each scanning point on the electron beam scanning plane are simulated according to the working principle of scanning electron microscope to obtain the corresponding first / second electron microscope images. Then, pixel-level feature fusion is performed on the first / second electron microscope images and the first visual image respectively to obtain the corresponding first / second electron microscope image fused image. The embodiments of this invention can generate strictly registered visual images and electron microscope images, bridging the cognitive gap between intuitive morphology and physical signals; the embodiments of this invention construct an end-to-end fully automated pipeline process from geometric modeling to attitude control to multiphysics simulation, solving the problem of process fragmentation and providing core technical support for high-throughput simulation; the embodiments of this invention provide a pixel-level fusion framework for physical signals and visual features, generating information-enhanced fused images by fusing BSE / SE signals with geometric features such as normals and depth. Attached Figure Description
[0013] Figure 1 This is a schematic diagram of a processing method for simulating scanning electron microscope images provided in Embodiment 1 of the present invention; Figure 2 This is a side view of the sample model provided in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the sample model rotation provided in Embodiment 1 of the present invention; Figure 4 This is a side view of a closed environment space provided in Embodiment 1 of the present invention; Figure 5This is a schematic diagram showing the positions of the light source point, camera point, and camera imaging plane provided in Embodiment 1 of the present invention. Figure 6 This is a schematic diagram showing the positions of the electron gun and the electron beam scanning plane provided in Embodiment 1 of the present invention; Figure 7 This is a module structure diagram of a processing device for simulating scanning electron microscope images provided in Embodiment 2 of the present invention; Figure 8 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. Detailed Implementation
[0014] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.
[0015] Embodiment 1 of the present invention provides a processing method for simulating scanning electron microscope images, such as... Figure 1 The schematic diagram shows a processing method for simulating scanning electron microscope images provided in Embodiment 1 of the present invention. The method mainly includes the following steps: Step 1: Receive the simulated task data input by the user.
[0016] Here, the simulation task data in this embodiment of the invention includes a sample generation mode and modeling reference data; wherein, the sample generation mode includes a first mode and a second mode; when the sample generation mode is the first mode, the modeling reference data is a three-dimensional voxel file; when the sample generation mode is the second mode, the modeling reference data is a set of generative geometric parameters.
[0017] Step 2: Based on the simulated task data, perform sample voxel structure modeling to obtain the corresponding sample model; and based on the Marching Cubes algorithm, extract the triangular mesh network from the sample surface of the sample model to obtain the corresponding surface mesh network; Specifically, this includes: Step 21, modeling the sample voxel structure based on the simulated task data to obtain the corresponding sample model; Specifically, it includes: step 211, identifying the sample generation mode; if the sample generation mode is the first mode, then the three-dimensional voxel file of the modeling reference data is used as the corresponding voxel set V; if the sample generation mode is the second mode, then the three-dimensional structure is modeled based on the generative geometric parameters of the modeling reference data, and the voxel data of the current three-dimensional structure is extracted and the extracted data is used to form the corresponding voxel set V. Here, the generative geometric parameters of this embodiment of the invention include at least fiber structure geometric parameters and carbon fiber structure geometric parameters; The voxel set V in this embodiment of the invention includes multiple voxel points v; each voxel point v includes voxel coordinates and voxel feature values; Step 212: Binarize all voxel feature values of voxel set V, reset voxel feature values greater than or equal to the preset voxel feature threshold to 1, and reset voxel feature values less than the voxel feature threshold to 0. Here, the voxel feature threshold in this embodiment of the invention is a pre-set threshold parameter; Step 213: Construct the voxel mesh space corresponding to the voxel set V in the XYZ coordinate system and denote it as the current mesh space; and in the current mesh space, first denoise the voxel set V based on the morphological opening operation principle, and then fill the holes in the denoised voxel set V based on the morphological closing operation principle. Here, the shape of the voxel grid space in this embodiment of the invention is a cuboid shape; Step 214: Based on the preset sample 3D size, randomly cut out a subspace from the current grid space as the corresponding sample grid space; and form a sample model from all voxel points v corresponding to the sample grid space. Here, the three-dimensional dimensions of the sample in this embodiment of the invention are a set of pre-set length, width, and height three-dimensional dimension data; the three-dimensional dimensions of the sample grid space in this embodiment of the invention satisfy the three-dimensional dimensions of the sample; Step 22, and extract the corresponding surface mesh network by performing triangular mesh network extraction on the sample surface of the sample model based on the Marching Cubes algorithm.
[0018] Here, the surface mesh network and sample substrate surface of the sample model in this embodiment of the invention are as follows: Figure 2 The side view of the sample model provided in Embodiment 1 of the present invention is shown.
[0019] Step 3: Translate the sample model so that its base surface is on the XOY plane of the XYZ coordinate system and the projection center of the sample model on the XOY plane is the origin O of the coordinate system; rotate the translated sample model according to the preset X-axis and Y-axis rotation angles; and model the closed environment space that can surround the sample model after the rotation is completed.
[0020] Here, the rotation angles of the X-axis and Y-axis in this embodiment of the invention are two preset rotation angles. By default, the rotation is first along the X-axis and then along the Y-axis.
[0021] The position of the model after translation of the sample model, such as Figure 3The diagram shows a sample model rotation provided in Embodiment 1 of the present invention; the model position after pose rotation of the sample model is as follows: Figure 3 As shown. It should be noted that, in this embodiment of the invention, when translating or rotating the sample model, the surface mesh network will be translated and rotated synchronously.
[0022] The spatial inclusion relationship between the enclosed environment space and the sample model in the embodiments of the present invention is as follows: Figure 4 The image shows a side view of the enclosed environment space provided in Embodiment 1 of the present invention. The spatial structure of this enclosed environment space is a hexagonal cuboid structure.
[0023] Step 4: Set up light source points on the positive Z-axis above the sample model and set the light source intensity of the light source points; set up camera points on the positive Z-axis above the sample model; set up a camera imaging plane between the camera points and the sample model; and simulate the normals, depths and RGB features of each pixel on the camera imaging plane according to the orthogonal camera imaging principle to obtain the corresponding first visual image. Specifically, this includes: Step 41, setting light source points on the positive Z-axis above the sample model, setting the light source intensity of the light source points, setting camera points on the positive Z-axis above the sample model, and setting a camera imaging plane between the camera points and the sample model; Here, the positional relationship between the light source point, the camera point, and the camera imaging plane in this embodiment of the invention is as follows: Figure 5 The diagram shows the positions of the light source point, camera point, and camera imaging plane provided in Embodiment 1 of the present invention; the camera imaging plane of this embodiment is parallel to the XOY plane. Step 42, and simulate the normals, depths and RGB features of each pixel on the camera imaging plane according to the orthogonal camera imaging principle to obtain the corresponding first visual image; Specifically, it includes: Step 421, according to the parallel projection principle of orthogonal camera imaging, confirming and recording the corresponding surface points of each pixel on the camera imaging plane on the surface grid network as the corresponding first surface points; Step 422: Calculate the normal direction of each first surface point on the surface mesh network to obtain the corresponding first normal direction; Step 423: Calculate the direction of the light rays from the point light source to each first surface point to obtain the corresponding first ray direction; Step 424: According to the Lambertian diffuse reflection model, calculate the corresponding first RGB pixel value of the current surface point based on the light source intensity and the first normal direction and first ray direction of each first surface point. Step 425: Calculate the vertical distance from each first surface point to the camera imaging plane and use the calculation result as the corresponding first depth; Step 426: The first pixel feature is composed of the first normal direction, the first depth and the first RGB pixel value corresponding to each first surface point; Step 427: The first visual image is composed of the first pixel features of all pixels on the camera imaging plane.
[0024] Step 5: Set up the electron gun on the positive Z-axis above the sample model; set the electron beam scanning plane of the electron gun to a spatial plane with the same spatial position, size and grid resolution as the camera imaging plane, ensuring that the scanning points of the electron beam correspond one-to-one with the pixels of the visual image; set the corresponding BSE and SE detector signal receiving ranges on the electron beam scanning plane; and simulate the signal characteristics of backscattered electrons and secondary electrons at each scanning point on the electron beam scanning plane according to the working principle of scanning electron microscope to obtain the corresponding first electron microscope image and second electron microscope image. Specifically, this includes: Step 51, setting up the electron gun on the positive Z-axis above the sample model; setting the electron beam scanning plane of the electron gun to a spatial plane with the same spatial position, size and grid resolution as the camera imaging plane, ensuring that the scanning points of the electron beam correspond one-to-one with the pixels of the visual image; and setting the corresponding BSE and SE detector signal receiving ranges on the electron beam scanning plane. Here, the positional relationship between the electron gun and the electron beam scanning plane in this embodiment of the invention is as follows: Figure 6 The diagram shows the positions of the electron gun and the electron beam scanning plane provided in Embodiment 1 of the present invention; In this embodiment of the invention, the scanning points on the electron beam scanning plane correspond one-to-one with the pixels on the camera imaging plane; The signal receiving range of the BSE / SE detector in this embodiment of the invention is two independent plane configuration ranges, and the electron beam scanning plane can be used as the configuration range at the maximum. The specific range can be set according to the actual experimental requirements, and no specific limitation is made here. Step 52, and according to the working principle of scanning electron microscope, simulate the signal characteristics of backscattered electrons and secondary electrons at each scanning point on the electron beam scanning plane to obtain the corresponding first electron microscope image and second electron microscope image; Specifically, this includes: step 521, setting the energy of the electron beam emitted by the electron gun to each scanning point; Here, the electron gun simulated in this embodiment of the invention scans each scanning point of the electron beam scanning plane point by point; and during each point scan, a corresponding electron beam is emitted towards the current scanning point; in this embodiment of the invention, the incident direction of all electron beams is perpendicular to the electron beam scanning plane; in this embodiment of the invention, the electron beams correspond one-to-one with the scanning points of the electron beam scanning plane; Step 522: Using the Monte Carlo simulation method, simulate the trajectory of each electron in each electron beam after entering the sample model; Step 523: Based on the electron simulation trajectory, the number of electrons that reach the signal receiving range of the BSE detector in each electron beam is counted to obtain the corresponding BSE signal strength. Here, the BSE signal intensity corresponding to each electron beam is also the BSE signal intensity corresponding to each scan point; Step 524: Based on the electron simulation trajectory, the number of electrons that reach the SE detector signal receiving range in each electron beam is counted to obtain the corresponding SE signal intensity. Here, the SE signal intensity corresponding to each electron beam is also the SE signal intensity corresponding to each scan point; Step 525: Perform grayscale conversion on the BSE signal intensity corresponding to each scanning point to obtain the corresponding first pixel grayscale; and use all the obtained first pixel grayscales to form the corresponding first electron microscope image; Step 526: Perform grayscale conversion on the SE signal intensity corresponding to each scanning point to obtain the corresponding second pixel grayscale; and use all the obtained second pixel grayscales to form the corresponding second electron microscope image.
[0025] Step 6: Perform pixel-level feature fusion of the first and second electron microscope images with the first visual image to obtain the corresponding first fused image and second fused image; and provide the current user with a corresponding first simulation report composed of the sample model, surface mesh network, first visual image, first electron microscope image, second electron microscope image, first fused image and second fused image.
[0026] Figure 7 This is a module structure diagram of a processing device for simulating scanning electron microscope images provided in Embodiment 2 of the present invention. This device can be a terminal device or server implementing the aforementioned method embodiments, or it can be a device that enables the aforementioned terminal device or server to implement the aforementioned method embodiments. For example, the device can be a device or chip system of the aforementioned terminal device or server. Figure 7 As shown, the device includes: a data receiving module 201, a sample voxel and sample surface modeling module 202, a sample pose adjustment and sample environment modeling module 203, a visual imaging rendering module 204, a scanning electron microscope image simulation module 205, and a multimodal fusion module 206.
[0027] The data receiving module 201 is used to receive simulation task data input by the user; the simulation task data includes sample generation mode and modeling reference data; the sample generation mode includes a first mode and a second mode; when the sample generation mode is the first mode, the modeling reference data is a three-dimensional voxel file; when the sample generation mode is the second mode, the modeling reference data is a set of generative geometric parameters.
[0028] The sample voxel and sample surface modeling module 202 performs sample voxel structure modeling based on simulation task data to obtain the corresponding sample model; and extracts the corresponding surface mesh network by extracting the triangular mesh network of the sample model based on the Marching Cubes algorithm.
[0029] The sample pose adjustment and sample environment modeling module 203 is used to translate the sample model so that its base surface is on the XOY plane of the XYZ coordinate system and the projection center of the sample model on the XOY plane is the origin O of the coordinate system; and to rotate the translated sample model according to the preset X-axis and Y-axis rotation angles; and to model the closed environment space that can surround the sample model after the rotation is completed; wherein, when translating and rotating the sample model, the surface mesh network is translated and rotated synchronously; the spatial structure of the closed environment space is a six-sided cuboid structure.
[0030] The visual imaging rendering module 204 is used to set light source points on the positive Z-axis above the sample model and set the light source intensity of the light source points, and set camera points on the positive Z-axis above the sample model; and set a camera imaging plane between the camera points and the sample model; and simulate the normal, depth and RGB features of each pixel on the camera imaging plane according to the orthogonal camera imaging principle to obtain the corresponding first visual image; wherein, the camera imaging plane is parallel to the XOY plane.
[0031] The scanning electron microscope (SEM) image simulation module 205 is used to set the electron gun on the positive Z-axis above the sample model; and to set the electron beam scanning plane of the electron gun to a spatial plane with the same spatial position, size and grid resolution as the camera imaging plane, ensuring that the scanning points of the electron beam correspond one-to-one with the pixels of the visual image; and to set the corresponding BSE and SE detector signal receiving ranges on the electron beam scanning plane; and to simulate the signal characteristics of backscattered electrons and secondary electrons at each scanning point on the electron beam scanning plane according to the working principle of the SEM to obtain the corresponding first and second SEM images; the scanning points on the electron beam scanning plane correspond one-to-one with the pixels of the camera imaging plane.
[0032] The multimodal fusion module 206 is used to perform pixel-level feature fusion on the first and second electron microscope images and the first visual image respectively to obtain the corresponding first fusion image and second fusion image; and the first simulation report composed of the sample model, surface mesh network, first visual image, first electron microscope image, second electron microscope image, first fusion image and second fusion image is fed back to the current user.
[0033] The processing apparatus for simulating scanning electron microscope images provided in this embodiment of the invention can execute the method steps in the above method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.
[0034] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented in software via processing element calls, while others are implemented in hardware. For example, the data receiving module can be a separate processing element, or it can be integrated into a chip in the above device. Alternatively, it can be stored as program code in the memory of the above device, and called and executed by a processing element of the device. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.
[0035] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a System-on-a-Chip (SOC).
[0036] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the foregoing method embodiments are generated. The computer described above can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The aforementioned computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the aforementioned computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, Bluetooth, microwave, etc.) means. The aforementioned computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).
[0037] Figure 8 This is a schematic diagram of an electronic device provided in Embodiment 3 of the present invention. This electronic device can be a terminal device or server implementing the methods of the aforementioned embodiments, or it can be a terminal device or server connected to the aforementioned terminal device or server implementing the methods of the aforementioned embodiments. Figure 8 As shown, the electronic device may include: a processor 301 (e.g., CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transmission and reception operations of the transceiver 303. The memory 302 may store various instructions for performing various processing functions and implementing the processing steps described in the foregoing embodiments. Preferably, the electronic device involved in the embodiments of the present invention further includes: a power supply 304, a system bus 305, and a communication port 306. The system bus 305 is used to realize communication connections between components. The communication port 306 is used for communication between the electronic device and other peripherals.
[0038] exist Figure 8The system bus 305 mentioned can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, it is represented by only one thick line in the figure, but this does not indicate that there is only one bus or one type of bus. The communication interface is used to enable communication between the database access device and other devices (e.g., clients, read-write libraries, and read-only libraries). Memory may include Random Access Memory (RAM) and may also include Non-Volatile Memory, such as at least one disk storage device.
[0039] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), graphics processing units (GPUs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0040] It should be noted that the embodiments of the present invention also provide a computer-readable storage medium storing instructions that, when run on a computer, cause the computer to perform the methods and processes provided in the above embodiments.
[0041] This invention provides a processing method, apparatus, electronic device, and computer-readable storage medium for simulating scanning electron microscope (SEM) images. As described above, this invention can directly obtain a sample model by modeling the sample voxel structure from a 3D voxel file, or it can first perform 3D modeling and voxel file extraction based on generative geometric parameters, and then obtain a sample model based on the voxel file. After obtaining the sample model, a surface mesh network is obtained by extracting the triangular mesh network of the sample surface using the Marching Cubes algorithm. Then, the sample model and the surface mesh network are synchronously translated and rotated, and after pose adjustment, a closed environment space (a six-sided cuboid structure) that surrounds the sample model is modeled. Then, light source points are set on the positive Z-axis above the sample model, and camera points are set on the positive Z-axis above the model. Finally, the normals, depths, and RGB features of each pixel on the camera imaging plane are simulated according to the orthogonal camera imaging principle to obtain a first visual image. Then, the electron gun is set on the positive Z-axis above the sample model; the electron beam scanning plane of the electron gun is set to a spatial plane with the same spatial position, size, and grid resolution as the camera imaging plane, ensuring a one-to-one correspondence between the scanning points of the electron beam and the pixels of the visual image; the corresponding BSE and SE detector signal receiving ranges are set on the electron beam scanning plane; and the BSE / SE signal characteristics of each scanning point on the electron beam scanning plane are simulated according to the working principle of scanning electron microscope to obtain the corresponding first / second electron microscope images. Then, pixel-level feature fusion is performed on the first / second electron microscope images and the first visual image respectively to obtain the corresponding first / second electron microscope image fused image. The embodiments of this invention can generate strictly registered visual images and electron microscope images, bridging the cognitive gap between intuitive morphology and physical signals; the embodiments of this invention construct an end-to-end fully automated pipeline process from geometric modeling to attitude control to multiphysics simulation, solving the problem of process fragmentation and providing core technical support for high-throughput simulation; the embodiments of this invention provide a pixel-level fusion framework for physical signals and visual features, generating information-enhanced fused images by fusing BSE / SE signals with geometric features such as normals and depth.
[0042] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0043] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A processing method for simulating scanning electron microscope images, characterized in that, The method includes: The system receives simulation task data input by the user; the simulation task data includes a sample generation mode and modeling reference data; the sample generation mode includes a first mode and a second mode; when the sample generation mode is the first mode, the modeling reference data is a three-dimensional voxel file; when the sample generation mode is the second mode, the modeling reference data is a set of generative geometric parameters. Based on the simulated task data, sample voxel structure modeling is performed to obtain the corresponding sample model; and based on the MarchingCubes algorithm, triangular mesh network is extracted from the sample surface of the sample model to obtain the corresponding surface mesh network. The sample model is translated so that its base surface is on the XOY plane of the XYZ coordinate system, and the center of the projection of the sample model on the XOY plane is the origin O of the coordinate system. The translated sample model is then rotated according to preset X-axis and Y-axis rotation angles. After the rotation, a closed environment space enclosing the sample model is modeled. During the translation and rotation of the sample model, the surface mesh network is simultaneously translated and rotated. The spatial structure of the closed environment space is a hexagonal cuboid. A light source point is set on the positive Z-axis above the sample model, and the light intensity of the light source point is set. A camera point is also set on the positive Z-axis above the sample model. A camera imaging plane is set between the camera point and the sample model. The normal, depth, and RGB features of each pixel on the camera imaging plane are simulated according to the orthogonal camera imaging principle to obtain the corresponding first visual image. The camera imaging plane is parallel to the XOY plane. An electron gun is set on the positive Z-axis above the sample model; the electron beam scanning plane of the electron gun is set to a spatial plane with the same spatial position, size, and grid resolution as the camera imaging plane, ensuring that the scanning points of the electron beam correspond one-to-one with the pixels of the visual image; the corresponding BSE and SE detector signal receiving ranges are set on the electron beam scanning plane; and the signal characteristics of backscattered electrons and secondary electrons at each scanning point on the electron beam scanning plane are simulated according to the working principle of scanning electron microscope to obtain the corresponding first electron microscope image and second electron microscope image; the scanning points on the electron beam scanning plane correspond one-to-one with the pixels of the camera imaging plane. The first and second electron microscope images are respectively fused with the first visual image at the pixel level to obtain the corresponding first fused image and second fused image; and the first simulation report, composed of the sample model, the surface mesh network, the first visual image, the first electron microscope image, the second electron microscope image, the first fused image and the second fused image, is fed back to the current user.
2. The processing method for simulating scanning electron microscope images according to claim 1, characterized in that, The process of obtaining the corresponding sample model by modeling the sample voxel structure based on the simulated task data specifically includes: Step 21: Identify the sample generation mode; if the sample generation mode is the first mode, then use the three-dimensional voxel file of the modeling reference data as the corresponding voxel set V; if the sample generation mode is the second mode, then perform three-dimensional structure modeling based on the generative geometric parameters of the modeling reference data, and extract voxel data from the current three-dimensional structure and form the corresponding voxel set V from the extracted data. The generative geometric parameters include at least fiber structure geometric parameters and carbon fiber structure geometric parameters; the voxel set V includes multiple voxel points v; each voxel point v includes voxel coordinates and voxel feature values. Step 22: Binarize all the voxel feature values of the voxel set V, reset the voxel feature values that are greater than or equal to a preset voxel feature threshold to 1, and reset the voxel feature values that are less than the voxel feature threshold to 0; Step 23: Construct the voxel mesh space corresponding to the voxel set V in the XYZ coordinate system and denote it as the current mesh space; and in the current mesh space, first denoise the voxel set V based on the morphological opening operation principle, and then fill the holes in the denoised voxel set V based on the morphological closing operation principle; The shape of the voxel grid space is a cuboid. Step 24: Based on the preset sample 3D size, randomly cut out a subspace from the current grid space as the corresponding sample grid space; and form the sample model by all the voxel points v corresponding to the sample grid space. The three-dimensional dimensions of the sample grid space satisfy the three-dimensional dimensions of the sample.
3. The processing method for simulating scanning electron microscope images according to claim 1, characterized in that, The process of simulating the normals, depths, and RGB features of each pixel on the camera's imaging plane according to the orthogonal camera imaging principle to obtain the corresponding first visual image specifically includes: Step 31: According to the parallel projection principle of orthogonal camera imaging, the corresponding surface points of each pixel on the camera imaging plane on the surface grid network are identified and recorded as the corresponding first surface points. Step 32: Calculate the corresponding first normal direction for each first surface point on the surface mesh network; Step 33: Calculate the direction of the light rays from the point light source to each of the first surface points to obtain the corresponding first light ray direction; Step 34: According to the Lambertian diffuse reflection model, calculate the corresponding first RGB pixel value of the current surface point based on the light source intensity and the first normal direction and the first light direction of each first surface point. Step 35: Calculate the vertical distance from each of the first surface points to the camera imaging plane and use the calculation result as the corresponding first depth; Step 36: The first pixel feature is composed of the first normal direction, the first depth and the first RGB pixel value corresponding to each first surface point; Step 37: The first visual image is composed of the first pixel features of all pixels on the camera imaging plane.
4. The processing method for simulating scanning electron microscope images according to claim 1, characterized in that, The process of simulating the signal characteristics of backscattered electrons and secondary electrons at each scanning point on the electron beam scanning plane according to the working principle of scanning electron microscope to obtain the corresponding first electron microscope image and second electron microscope image specifically includes: Step 41: Set the energy of the electron beam emitted by the electron gun to each scanning point; The electron gun scans each scanning point on the electron beam scanning plane point by point; and emits a corresponding electron beam to the current scanning point during each scanning, with the incident direction of the electron beam perpendicular to the electron beam scanning plane; the electron beam corresponds one-to-one with the scanning point. Step 42: Using the Monte Carlo simulation method, simulate the trajectory of each electron in each electron beam after entering the sample model; Step 43: Based on the electron simulation trajectory, the number of electrons that reach the signal receiving range of the BSE detector in each electron beam is counted to obtain the corresponding BSE signal strength. Step 44: Based on the electron simulation trajectory, the number of electrons that reach the SE detector signal receiving range in each electron beam is counted to obtain the corresponding SE signal intensity. Step 45: Perform grayscale conversion on the BSE signal intensity corresponding to each scanning point to obtain the corresponding first pixel grayscale; and combine all the obtained first pixel grayscales to form the corresponding first electron microscope image; Step 46: Perform grayscale conversion on the SE signal intensity corresponding to each scanning point to obtain the corresponding second pixel grayscale; and use all the obtained second pixel grayscales to form the corresponding second electron microscope image.
5. An apparatus for performing the processing method for simulating scanning electron microscopy images according to any one of claims 1-4, characterized in that, The device includes: a data receiving module, a sample voxel and sample surface modeling module, a sample pose adjustment and sample environment modeling module, a visual imaging rendering module, a scanning electron microscope image simulation module, and a multimodal fusion module. The data receiving module is used to receive simulation task data input by the user; the simulation task data includes sample generation mode and modeling reference data; the sample generation mode includes a first mode and a second mode; when the sample generation mode is the first mode, the modeling reference data is a three-dimensional voxel file; when the sample generation mode is the second mode, the modeling reference data is a set of generative geometric parameters. The sample voxel and sample surface modeling module performs sample voxel structure modeling based on the simulation task data to obtain the corresponding sample model; and extracts the corresponding surface mesh network by performing triangular mesh network extraction on the sample surface of the sample model based on the Marching Cubes algorithm. The sample pose adjustment and sample environment modeling module is used to translate the sample model so that its base surface is on the XOY plane of the XYZ coordinate system, and the projection center of the sample model on the XOY plane is the origin O of the coordinate system; and to rotate the translated sample model according to preset X-axis and Y-axis rotation angles; and to model the closed environment space that can surround the sample model after the rotation; wherein, when translating and rotating the sample model, the surface mesh network is translated and rotated synchronously; the spatial structure of the closed environment space is a hexagonal cuboid structure; The visual imaging rendering module is used to set light source points on the positive Z-axis above the sample model, set the light intensity of the light source points, and set camera points on the positive Z-axis above the sample model; and set a camera imaging plane between the camera points and the sample model; and simulate the normals, depths, and RGB features of each pixel on the camera imaging plane according to the orthogonal camera imaging principle to obtain the corresponding first visual image; wherein, the camera imaging plane is parallel to the XOY plane; The scanning electron microscope (SEM) image simulation module is used to set the electron gun on the positive Z-axis above the sample model; and to set the electron beam scanning plane of the electron gun to a spatial plane with the same spatial position, size, and grid resolution as the camera imaging plane, ensuring that the scanning points of the electron beam correspond one-to-one with the pixels of the visual image; and to set the corresponding BSE and SE detector signal receiving ranges on the electron beam scanning plane; and to simulate the signal characteristics of backscattered electrons and secondary electrons at each scanning point on the electron beam scanning plane according to the working principle of the SEM to obtain the corresponding first and second SEM images; the scanning points on the electron beam scanning plane correspond one-to-one with the pixels of the camera imaging plane. The multimodal fusion module is used to perform pixel-level feature fusion of the first and second electron microscope images with the first visual image to obtain corresponding first fused image and second fused image; and the first simulation report composed of the sample model, the surface mesh network, the first visual image, the first electron microscope image, the second electron microscope image, the first fused image and the second fused image is fed back to the current user.
6. An electronic device, characterized in that, include: Memory, processor, and transceiver; The processor is configured to be coupled to the memory, read and execute instructions in the memory to implement the method according to any one of claims 1-4; The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.
7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1-4.