Target point rapid positioning method and system based on scanning electron microscope observation

Abstract

The embodiment of the invention provides a target point rapid positioning method and system based on scanning electron microscope observation. Wherein during the first observation, a target point is selected on a to-be-detected sample, a multi-layer reference marker system is established around the point, and the relative angle and the linear distance between the target point and a reference marker are determined by adjusting the angle of a sample table and recording observation conditions for acquiring a sample image each time; the position information of the reference marker in the three-dimensional coordinate system is reconstructed by using the angle difference and the position difference in the plurality of sample images, a three-dimensional space geometrical relationship network is formed, and when the sample is subjected to scanning electron microscope observation again after being processed, the initial position of the target point is calculated by using the three-dimensional space geometrical relationship network. And moving the sample stage at a sub-pixel level until the target point is located at the center of the view, and monitoring and correcting system errors or sample displacement. According to the technical scheme provided by the embodiment of the invention, the accuracy, adaptability and efficiency of target point positioning are improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a method and system for quickly locating a target point based on scanning electron microscope observation. Background Art

[0002] In the fields of materials science, biology and nanotechnology, scanning electron microscopy (SEM) is an important observation tool and is widely used for the analysis of microstructures.

[0003] At present, the common practice is to record the location coordinates of the target point during the first scan and record as much information about the surrounding environment as possible to facilitate subsequent repositioning. In addition, there is also a manual marking method, which is to add easily identifiable artificial markers to the sample to assist in positioning.

[0004] However, traditional positioning methods lack systematicity and stability. When the sample undergoes processing, the original coordinate information may become invalid; manual marking may introduce additional variables and interfere with the experimental results; existing positioning methods usually do not take into account the three-dimensional spatial position changes caused by the angle adjustment of the sample stage, which makes it difficult to relocate the target point under complex observation conditions. Summary of the invention

[0005] The embodiments of the present application provide a method and system for quickly locating a target point based on scanning electron microscope observation, so as to solve the problems of low efficiency, lack of accuracy and adaptability of target point positioning in the prior art.

[0006] In a first aspect, an embodiment of the present application provides a method for quickly locating a target point based on scanning electron microscope observation, comprising:

[0007] When the sample to be tested is observed by scanning electron microscopy for the first time, a target point is selected on the sample to be tested, and a multi-level, multi-scale reference marker system is established around the target point, and each level of the reference marker system contains at least three non-collinear morphological features as reference markers, so as to ensure that the target point selected by the sample to be tested at different magnifications has a corresponding reference marker system as a positioning reference;

[0008] By adjusting the angle of the sample stage on which the sample to be tested is placed, each time the angle is adjusted and a sample image including the target point and the reference markers around the target point is acquired through a scanning electron microscope, the observation conditions when the sample image is acquired are simultaneously recorded, including the observation angle, the sample stage angle, and the magnification of the scanning electron microscope;

[0009] Determining from each sample image the relative angle and linear distance between the target point and the reference markers selected at each level and between the reference markers;

[0010] Based on the multiple sample images, determine the angle difference and position difference of the same reference marker under different observation angles, and reconstruct the position information of the reference marker in a three-dimensional coordinate system based on the angle difference and position difference through stereo vision technology to form a three-dimensional spatial geometric relationship network;

[0011] When the sample to be tested is processed and subjected to a second scanning electron microscope observation, the reference markers of the corresponding level are determined according to the current magnification, and the position information of the reference markers of the current level and the relative angles and straight-line distances between the reference markers are obtained by using the three-dimensional spatial geometric relationship network, and the preliminary position of the target point is calculated;

[0012] A local fine-tuning algorithm is applied in combination with the sample image collected in real time. According to the preliminary position of the target point, the sample stage is moved at the sub-pixel level until the target point is located at the center of the field of view, so as to determine the target position of the target point. During the movement of the sample stage, possible system errors or sample displacements are monitored and corrected to ensure that the originally determined relative angles and straight-line distances between the reference markers remain consistent.

[0013] Optionally, determining the angle difference and position difference of the same reference marker under different observation angles based on the multiple sample images, and reconstructing the position information of the reference marker in a three-dimensional coordinate system based on the angle difference and position difference through stereo vision technology to form a three-dimensional space geometric relationship network includes:

[0014] For each sample image, a computer vision algorithm is used to extract feature points of the reference marker, and a descriptor is generated for each feature point, wherein the feature point refers to a key point that can uniquely identify the position and shape of the reference marker in the sample image;

[0015] By comparing feature descriptors in different sample images, feature points of the same reference marker from different observation angles are identified, and feature point pairs are generated, wherein the feature point pairs are composed of one or more pairs of feature points from different sample images, and the feature points in each feature point pair have similar feature descriptors and correspond to the same physical position;

[0016] Based on the feature point pairs, geometric constraints are applied to estimate the relative geometric relationship between different observation perspectives, and the rotation and translation changes of the camera on the extrinsic parameters are calculated by decomposing the relative geometric relationship to obtain the camera posture information;

[0017] According to the feature point pairs and the camera posture information, the relative rotation angle of the same reference marker at different observation angles is calculated as the angle difference, and the relative displacement of the same reference marker at different observation angles is calculated as the position difference;

[0018] An improved triangulation algorithm is applied, combined with the observation conditions and geometric constraints corresponding to each sample image, and the angle difference and position difference are used to reconstruct the position information of the reference marker in the three-dimensional coordinate system to form a three-dimensional spatial geometric relationship network.

[0019] Optionally, the application of an improved triangulation algorithm, in combination with observation conditions and geometric constraints corresponding to each sample image, utilizes the angle difference and position difference to reconstruct the position information of the reference marker in the three-dimensional coordinate system to form a three-dimensional spatial geometric relationship network, including:

[0020] Applying an improved triangulation algorithm, combining observation conditions and geometric constraints corresponding to each sample image, and utilizing the angle difference and the position difference, calculates position information of the feature point pair in a three-dimensional coordinate system;

[0021] By integrating the position information of all feature point pairs, the three-dimensional coordinates of multiple feature points on each reference marker are obtained;

[0022] Determine the position information of each reference marker by fitting the three-dimensional coordinates of a plurality of feature points of each reference marker, wherein the position information includes the three-dimensional coordinates and the posture;

[0023] Based on the position information of each reference marker, a three-dimensional spatial geometric relationship network is constructed to describe the relative angles and straight-line distances between all reference markers and between the reference markers and the target point. In the process of constructing the three-dimensional spatial geometric relationship network, the angle differences and position differences are used to correct possible errors.

[0024] Optionally, the using the three-dimensional spatial geometric relationship network to obtain the position information of the reference markers at the current level and the relative angles and straight-line distances between the reference markers, and calculating the preliminary position of the target point includes:

[0025] Using the three-dimensional spatial geometric relationship network, the position information of the reference markers and the relative angles between the reference markers under the current observation conditions are obtained to reproduce the observation conditions when the sample to be tested is first observed by the scanning electron microscope;

[0026] Recovering the spatial geometric relationship between the target point and the reference markers based on the relative angle and straight-line distance of the target point relative to the reference marker recorded under the observation conditions when the sample to be tested is first observed under the scanning electron microscope, combined with the position information of the reference markers under the current observation conditions and the relative angle and straight-line distance between the reference markers;

[0027] The spatial geometric relationship is used to calculate the preliminary position of the target point through an improved triangulation algorithm.

[0028] Optionally, the step of obtaining the position information of the reference markers and the relative angles between the reference markers under the current observation conditions to reproduce the observation conditions when the sample to be tested is first observed under the scanning electron microscope includes:

[0029] Based on the three-dimensional spatial geometric relationship network, obtaining the three-dimensional coordinates of each reference marker under the current observation condition as the position information of the reference marker;

[0030] Calculating the relative angles between all reference markers in the same layer according to the position information to determine the relative angles between the reference markers under the current observation condition;

[0031] The relative angle and straight-line distance between the reference markers and the angle of the sample stage under the current observation conditions are adjusted to ensure that the relative angle and straight-line distance between the reference markers are consistent with the relative angle and straight-line distance originally determined between the reference markers during the first observation, and the angle of the sample stage is consistent with the originally determined angle of the sample stage.

[0032] Optionally, it also includes:

[0033] When the sample to be tested is observed by a scanning electron microscope for the first time, multimodal data of the target point and the surrounding area are recorded and stored, wherein the multimodal data at least includes spectral characteristics and physical properties;

[0034] When the sample to be tested is processed and subjected to a second SEM observation, multimodal data of the current target point and the surrounding area are recollected;

[0035] By comparing the multimodal data stored during the first observation with the currently collected multimodal data, a multimodal feature matching result is generated, wherein the multimodal feature matching result includes change information of the target point and the surrounding area between the two observations;

[0036] Based on the multimodal feature matching result, the target position of the target point is adjusted to ensure the positioning accuracy of the target point without being affected by the microstructure change of the sample to be tested.

[0037] Optionally, the process of establishing a multi-level, multi-scale reference marker system around the target point, wherein each level of the reference marker system contains at least three non-collinear morphological features and serves as a reference marker includes: establishing a three-level reference marker system around the target point, and selecting in the first level of the reference marker system at least three non-collinear significant features of the overall structure of the sample to be tested on a macroscopic scale as reference markers, selecting in the second level of the reference marker system at least three non-collinear stable structures of the sample to be tested on a mesoscopic scale as reference markers, and selecting in the third level of the reference marker system at least three non-collinear fine features of the sample to be tested on a microscopic scale as reference markers.

[0038] Optionally, the geometric constraints include a fundamental matrix;

[0039] The method of applying the improved triangulation algorithm, combining the observation conditions and geometric constraints corresponding to each sample image, and using the angle difference and the position difference to calculate the position information of the feature point pair in the three-dimensional coordinate system includes:

[0040] The position information of the feature point pair in the three-dimensional coordinate system is calculated by the following calculation formula, wherein the position information includes the three-dimensional coordinates;

[0041]

[0042] w′ i =w i β i

[0043] w i =exp(-α·||T i || 2 -β·||Δp i || 2 )

[0044] β i =exp(γ·θ i )

[0045] Among them, P is the three-dimensional coordinate of the feature point, n is the number of feature point pairs, i is the index of the feature point pair, and w' i is the comprehensive weight of the i-th group of feature point pairs, K i is the camera intrinsic parameter matrix corresponding to the i-th set of feature point pairs, x 1,i and x 2,i are the 2D coordinates of the same reference marker in two different sample images, and F i is the basic matrix corresponding to the i-th group of feature point pairs, wi is the initial weight of the i-th group of feature point pairs, β iis the exponential adjustment factor of the i-th group of feature point pairs, θ is the angle difference, Δp is the position difference, α is a positive adjustment parameter used to control the influence of the translation vector, T i is the translation vector corresponding to the i-th set of feature point pairs, ||T i || represents the Euclidean norm of the translation vector, β is a positive adjustment parameter used to control the influence of position difference, Δp i is the position difference of the i-th group of feature point pairs, indicating the relative displacement between two different observation perspectives, ||Δp i || 2 is the Euclidean norm of the position difference, γ is a positive adjustment parameter used to control the influence of the rotation angle, θ i is the angle difference of the i-th set of feature point pairs, indicating the relative rotation angle between two different observation perspectives.

[0046] Optionally, the using the three-dimensional spatial geometric relationship network to obtain the position information of the reference markers at the current level and the relative angles and straight-line distances between the reference markers, and calculating the preliminary position of the target point includes:

[0047] Determine the position information of each reference marker in the three-dimensional coordinate system under the current observation conditions

[0048] Calculate relative angles between reference markers and straight-line distance

[0049] The linear distance d between the target point and the reference marker recorded under the observation conditions when the sample to be tested is first observed by the scanning electron microscope initial,ref , combined with the relative angles between the reference markers under the current observation conditions and straight-line distance Restore the spatial geometric relationship between the target point and the reference marker, and calculate the preliminary position p of the target point by the following formula initial :

[0050]

[0051] Where n is the number of reference markers; i is the index of the reference marker; is the weight of the i-th reference marker, according to the distance d from the target point to the reference marker under the first observation condition initial,ref The straight-line distance between the reference marker under the current observation conditions Determined, represents the straight-line distance between the i-th reference marker and all other reference markers j The sum of is the position information of the i-th reference marker in the three-dimensional coordinate system under the current observation conditions; ΔP i is the displacement vector of the target point relative to the i-th reference marker derived based on the spatial geometric relationship between the first observation condition and the current observation condition, and the displacement vector is calculated by the following formula:

[0052]

[0053] where θ initial,ij is the relative angle between the i-th reference marker and the j-th reference marker under the first observation condition; is the relative angle between the i-th reference marker and the j-th reference marker under the current observation condition; is the unit vector pointing from the i-th reference marker to the j-th reference marker.

[0054] In a second aspect, an embodiment of the present application provides a target point rapid positioning system based on scanning electron microscope observation, comprising:

[0055] A construction module is used to select a target point on the sample to be tested when the sample to be tested is observed by a scanning electron microscope for the first time, and to establish a multi-level, multi-scale reference marker system around the target point, and each level of the reference marker system contains at least three non-collinear morphological features as reference markers, so as to ensure that the target point selected by the sample to be tested at different magnifications has a corresponding reference marker system as a positioning reference;

[0056] an acquisition module, configured to adjust the angle of a sample stage on which the sample to be tested is placed, and each time the angle is adjusted and a sample image including the target point and reference markers around the target point is acquired through a scanning electron microscope, simultaneously record observation conditions when the sample image is acquired, including the angle of the sample stage and the magnification of the scanning electron microscope;

[0057] A determination module, for determining, from each sample image, the relative angle and linear distance between the target point and the reference markers selected at each level and between the reference markers;

[0058] A generation module, configured to determine the angle difference and position difference of the same reference marker under different observation angles based on the multiple sample images, and reconstruct the position information of the reference marker in a three-dimensional coordinate system based on the angle difference and position difference through stereo vision technology to form a three-dimensional spatial geometric relationship network;

[0059] A calculation module is used to determine the reference markers of the corresponding layer according to the current magnification when the sample to be tested is processed and subjected to a second scanning electron microscope observation, obtain the position information of the reference markers of the current layer and the relative angles between the reference markers by using the three-dimensional spatial geometric relationship network, and calculate the preliminary position of the target point;

[0060] The determination module is also used to apply a local fine-tuning algorithm, combined with a sample image collected in real time, and move the sample stage at a sub-pixel level according to the preliminary position of the target point until the target point is located at the center of the field of view, so as to determine the target position of the target point, and during the movement of the sample stage, by monitoring and correcting possible system errors or sample displacements, ensure that the originally determined relative angles and straight-line distances between the reference markers remain consistent.

[0061] In a third aspect, an embodiment of the present application provides a computing device, comprising a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement a method for rapid positioning of a target point based on scanning electron microscope observation as described in the first aspect above.

[0062] In a fourth aspect, an embodiment of the present application provides a computer storage medium storing a computer program, which, when executed by a computer, implements a method for quickly locating a target point based on scanning electron microscope observation as described in the first aspect.

[0063] In an embodiment of the present application, when a sample to be tested is observed by a scanning electron microscope for the first time, a target point is selected on the sample to be tested, and a multi-level, multi-scale reference marker system is established around the target point, and each level of the reference marker system contains at least three non-collinear morphological features and serves as a reference marker, so as to ensure that the target point selected by the sample to be tested at different magnifications has a corresponding reference marker system as a positioning reference; by adjusting the angle of the sample stage on which the sample to be tested is placed, each time the angle is adjusted and a sample image containing the target point and the reference markers around the target point is obtained through a scanning electron microscope, the observation conditions when the sample image is obtained are simultaneously recorded, including the observation viewing angle, the sample stage angle, and the magnification of the scanning electron microscope; the relative angle and straight-line distance between the target point and the reference markers selected at each level and between the reference markers are determined from each sample image; based on the multiple sample images, the position of the same reference marker at different viewing angles is determined. The angle difference and position difference under the measuring viewing angle are measured, and the position information of the reference marker in the three-dimensional coordinate system is reconstructed based on the angle difference and position difference through stereo vision technology to form a three-dimensional space geometric relationship network; when the sample to be measured is processed and subjected to a second scanning electron microscope observation, the reference marker of the corresponding level is determined according to the current magnification, and the position information of the reference marker of the current level and the relative angle and straight-line distance between the reference markers are obtained by using the three-dimensional space geometric relationship network, and the preliminary position of the target point is calculated; a local fine-tuning algorithm is applied, combined with the sample image collected in real time, and according to the preliminary position of the target point, the sample stage is moved at a sub-pixel level until the target point is located at the center of the field of view to determine the target position of the target point, and during the movement of the sample stage, the possible system errors or sample displacements are monitored and corrected to ensure that the originally determined relative angles and straight-line distances between the reference markers remain consistent.

[0064] The technical solution of this application has the following beneficial effects:

[0065] The present application selects the target point during the first SEM observation and establishes a multi-level, multi-scale reference marker system around it, ensuring that there is a reliable positioning reference under different magnifications. This method records the observation conditions after each adjustment of the sample stage angle in detail, and uses stereoscopic vision technology to reconstruct the position information of the reference marker in the three-dimensional coordinate system to form an accurate spatial geometric relationship network. When the sample to be tested needs to be observed again after processing, the reference marker of the corresponding level can be quickly determined according to the current magnification, and the preliminary position of the target point can be calculated in combination with the pre-constructed three-dimensional spatial geometric relationship network. Subsequently, a local fine-tuning algorithm is applied to perform sub-pixel-level precise positioning until the target point is accurately located at the center of the field of view. The entire process monitors and corrects possible system errors or sample displacements in real time to ensure the consistency and accuracy of the relative positions between reference markers. This method significantly improves the speed and accuracy of target point repositioning in the SEM, solves the problem of repeated positioning caused by sample changes that traditional methods are difficult to cope with, and can still maintain high efficiency and accuracy under high-resolution conditions. It not only enhances the stability and reliability of positioning, but also supports the analysis of complex structures, providing a powerful tool for research in fields such as materials science, biology and nanotechnology. By achieving fast and accurate target point positioning, this method greatly improves the work efficiency of microstructure analysis, simplifies the experimental process, and lays a solid foundation for subsequent research and application.

[0066] Furthermore, the embodiment of the present application also uses a computer vision algorithm to extract feature points and generate descriptors for reference markers in multiple sample images, and identifies feature point pairs at the same physical position by comparing feature descriptors under different observation angles. Based on these feature point pairs, geometric constraints are applied to estimate the relative geometric relationship between different observation angles, and the camera posture information is calculated. Furthermore, through the improved triangulation algorithm combined with observation conditions and geometric constraints, the position information of the reference marker in the three-dimensional coordinate system is reconstructed using angle differences and position differences to form an accurate three-dimensional spatial geometric relationship network. In this process, the position information of all feature points is also integrated to determine the three-dimensional coordinates and posture of each reference marker, and a three-dimensional spatial geometric relationship network that describes the relative relationship between all reference markers and between reference markers and target points is constructed, while correcting possible errors.

[0067] Through the above method, the accuracy and reliability of target point positioning in the scanning electron microscope are significantly enhanced. By accurately extracting and matching feature points, and accurately calculating the geometric relationship between different observation angles, it is ensured that a high-precision three-dimensional spatial geometric relationship network can be reconstructed even under complex conditions. The application of the improved triangulation algorithm enables more accurate determination of the three-dimensional position and posture of the reference marker, thereby improving the accuracy of the entire positioning system. In addition, the real-time correction of errors when constructing the three-dimensional spatial geometric relationship network further ensures the stability and consistency of the positioning results, providing a solid foundation for the subsequent precise positioning of target points. This method not only improves work efficiency, but also makes microstructure analysis more convenient and efficient, bringing important value to scientific research and technological applications.

[0068] These and other aspects of the present application will become more clearly understood in the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0070] Figure 1 A flowchart of a method for quickly locating a target point based on scanning electron microscope observation provided by the present application is shown;

[0071] Figure 2 A schematic diagram of the structure of a target point rapid positioning system based on scanning electron microscope observation provided by the present application is shown;

[0072] Figure 3 A schematic diagram of the structure of a computing device provided by the present application is shown. DETAILED DESCRIPTION

[0073] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0074] In some of the processes described in the specification and claims of this application and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0075] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0076] Figure 1 A flowchart of a method for quickly locating a target point based on scanning electron microscope observation is provided for an embodiment of the present application. Figure 1 As shown, the method includes:

[0077] 101. When the sample to be tested is observed by scanning electron microscopy for the first time, a target point is selected on the sample to be tested, and a multi-level, multi-scale reference marker system is established around the target point, and each level of the reference marker system contains at least three non-collinear morphological features as reference markers, so as to ensure that the target point selected by the sample to be tested at different magnifications has a corresponding reference marker system as a positioning reference;

[0078] Among them, the target point refers to a specific location selected on the sample to be tested with clear characteristics and research value. It is the core object of subsequent observation and analysis.

[0079] The multi-level, multi-scale reference marker system refers to the positioning benchmarks established at different magnifications; multi-scale means that these benchmarks include various morphological features from macro to micro. The three non-collinear morphological features in each level ensure that the target point can be stably found through these unchanging reference points even at different magnifications.

[0080] The process of establishing a multi-level, multi-scale reference marker system around the target point, wherein each level of the reference marker system contains at least three non-collinear morphological features as reference markers, includes: establishing a three-level reference marker system around the target point, and selecting at least three non-collinear significant features of the overall structure of the sample to be tested (such as edge corners, cracks, etc.) on a macro scale as reference markers in the first level of the reference marker system, selecting at least three non-collinear stable structures of the sample to be tested (such as pores, mineral particles) on a meso scale as reference markers in the second level of the reference marker system, and selecting at least three non-collinear fine features of the sample to be tested (such as crystal defects) on a micro scale as reference markers in the third level of the reference marker system.

[0081] In practice, a target point with clear characteristics is selected, and a series of markers with distinct morphological features are selected around it. These markers need to have sufficient contrast and stability to be recognizable at different magnifications. In this way, even if the sample is processed or the surface characteristics change, the target point can be found again based on these unchanging reference markers.

[0082] For example, in the field of artificial intelligence, machine learning algorithms can be used to automatically identify and annotate these reference markers. The training model can learn the representation of different morphological features through a large amount of annotated SEM image data, so as to automatically select appropriate reference markers on new samples. This not only improves efficiency, but also reduces human errors.

[0083] 102. By adjusting the angle of the sample stage on which the sample to be tested is placed, each time the angle is adjusted and a sample image including the target point and the reference markers around the target point is acquired through a scanning electron microscope, the observation conditions when the sample image is acquired are simultaneously recorded.

[0084] Whenever the sample stage angle changes, the system automatically acquires a new sample image and records the observation conditions at that time, which at least include the observation angle, sample stage angle, and SEM magnification. This information will be stored together with the image so that the observation environment at that time can be accurately reproduced in subsequent analysis.

[0085] Among them, the observation angle refers to the spatial position and direction of the scanning electron microscope relative to the sample to be tested, which determines the direction of imaging and the content of the acquired image. Different angles of view can reveal different sides or internal structures of the sample. The angle of the sample stage refers to the posture of the sample stage relative to a fixed reference system (such as a laboratory coordinate system), which is used to change the observation direction of the sample to obtain image information from more angles. Magnification is one of the working parameters of the scanning electron microscope, which determines the size of the sample area and the degree of detail displayed in the image. High magnification can show finer structures, while low magnification covers a larger area.

[0086] Continuing with the above example, the AI ​​system can integrate an automated control module to automatically rotate the sample stage and capture images according to a preset angle sequence. At the same time, sensor technology is used to monitor and record observation conditions in real time, such as temperature, humidity and other environmental factors, to ensure the consistency of experimental conditions. In addition, a deep learning model can be introduced to predict the best observation angle and optimize the image acquisition process.

[0087] 103. Determine, from each sample image, the relative angle and linear distance between the target point and the reference markers selected at each level and between the reference markers;

[0088] The relative angle refers to the angular relationship between the target point and the reference marker and between the reference markers, and is an important parameter for constructing a geometric model.

[0089] The straight-line distance is the actual physical distance between the target point and the reference marker and between the reference markers, which is used to accurately describe their spatial position relationship.

[0090] By analyzing each sample image, the relative position relationship between the target point and the selected reference marker, including angle and distance, is calculated. This process may involve image processing technology and computer vision algorithms.

[0091] At this stage of the above embodiment, advanced computer vision algorithms, such as SI FT (Scale Invariant Feature Transform) or ORB (Oriented FAST and Rotated BRI EF), can be applied to detect and match feature points and calculate the precise geometric relationship. Combined with deep learning methods, this process can be made more efficient and accurate.

[0092] 104. Determine the angle difference and position difference of the same reference marker under different observation angles based on the multiple sample images, and reconstruct the position information of the reference marker in the three-dimensional coordinate system based on the angle difference and position difference through stereo vision technology to form a three-dimensional space geometric relationship network;

[0093] Angle difference and position difference refer to the change in rotation angle and spatial displacement of the same reference marker under different observation perspectives, reflecting the geometric transformation between different observation perspectives.

[0094] Stereo vision technology uses two or more images from different observation perspectives to reconstruct a three-dimensional scene. By comparing the differences in images from different observation perspectives, the true three-dimensional coordinates of the object are calculated.

[0095] The three-dimensional spatial geometric relationship network is a three-dimensional coordinate system composed of all reference markers and their relationships with each other, which provides a complete description of the sample structure and supports subsequent precise positioning.

[0096] By using the feature point pairs in the multi-view images, the true three-dimensional coordinates of each reference marker are estimated through geometric methods such as triangulation, and a spatial geometric network describing the relationship between all reference markers is constructed.

[0097] For example, in the field of AI, technologies such as structured light scanning, binocular vision, or multi-view geometry can be used to achieve high-precision 3D reconstruction. In addition, traditional triangulation algorithms can be improved by training neural network models to improve the accuracy and speed of reconstruction. For example, convolutional neural networks (CNNs) can be used to extract more robust feature points to better handle noise and complex backgrounds.

[0098] 105. When the sample to be tested is processed and subjected to a second scanning electron microscope observation, the reference markers at the corresponding level are determined according to the current magnification, and the position information of the reference markers at the current level and the relative angles and straight-line distances between the reference markers are obtained by using the three-dimensional spatial geometric relationship network, and the preliminary position of the target point is calculated;

[0099] The preliminary position is the approximate position of the target point estimated based on the current observation conditions and the existing three-dimensional geometric relationship network, providing a starting point for the next step of precise positioning.

[0100] Based on the pre-established three-dimensional spatial geometric relationship network and the current observation conditions, the area where the target point is located is quickly locked and its approximate position is calculated. This step simplifies the subsequent fine-tuning process and improves positioning efficiency.

[0101] Based on the above embodiments, the AI ​​system can intelligently select the closest reference marker layer according to the current observation conditions, and quickly estimate the location of the target point by comparing the real-time image with the model in the database. The selection strategy is continuously optimized using machine learning algorithms, making the positioning process fast and accurate.

[0102] 106. Apply a local fine-tuning algorithm, in combination with a sample image collected in real time, and according to the preliminary position of the target point, move the sample stage at a sub-pixel level until the target point is located at the center of the field of view, so as to determine the target position of the target point. During the movement of the sample stage, monitor and correct possible system errors or sample displacements to ensure that the originally determined relative angles and straight-line distances between the reference markers remain consistent.

[0103] Local Fine Tuning Algorithm An optimization algorithm used to make small adjustments to the sample stage to achieve precise control at the sub-pixel level, ensuring that the target point is ultimately located in the center of the field of view.

[0104] Sub-pixel level refers to a level of accuracy smaller than a pixel and is commonly used in image processing and machine vision to achieve higher positioning accuracy.

[0105] Systematic error or sample displacement refers to measurement deviation or movement of the sample itself caused by factors such as wear of mechanical parts and temperature changes, which needs to be monitored and corrected in real time to ensure positioning accuracy.

[0106] Based on the preliminary positioning, the position of the sample stage is further adjusted to make the target point accurately in the center of the field of view. In this process, it is necessary to constantly compare the real-time image with the expected position and gradually reduce the error until the requirements are met.

[0107] On the basis of the above embodiments, by introducing a reinforcement learning algorithm, the AI ​​system can autonomously learn how to optimally perform sub-pixel level adjustments. The system will continuously optimize the adjustment strategy based on the feedback of the effect after each adjustment to achieve the best positioning effect. At the same time, a closed-loop control system is used to achieve real-time error correction to ensure the stability and reliability of the entire positioning process. In this way, from the initial reference marker selection to the final precise positioning of the target point, the entire process can be completed by an intelligent AI system, greatly improving work efficiency and accuracy.

[0108] Through the implementation of steps 101 to 106, the speed, accuracy and reliability of target point positioning in the scanning electron microscope are significantly improved. First, a multi-level and multi-scale reference marker system is established during the first observation (step 101), ensuring that the target points have reliable positioning benchmarks at different magnifications, providing a solid foundation for subsequent observations. Then, by adjusting the angle of the sample stage and recording detailed observation conditions (step 102), not only rich sample image data is obtained, but also the consistency and repeatability of each observation are guaranteed. On this basis, the relative geometric relationship between the target point and the reference marker is determined (step 103), laying the foundation for building an accurate spatial model. Subsequently, the three-dimensional coordinates of the reference marker are reconstructed using stereo vision technology and an improved triangulation algorithm (step 104), forming a complete spatial geometric relationship network. This step greatly improves the stability and accuracy of the positioning system, allowing high-precision three-dimensional reconstruction to be achieved even under complex conditions. When the sample needs to be observed again after processing, the corresponding reference marker layer is selected according to the current magnification, and the preliminary position of the target point is calculated using the existing three-dimensional geometric relationship network (step 105). This process quickly narrows the search range and greatly simplifies the final precise positioning process. Finally, the local fine-tuning algorithm is applied to move the sample stage at the sub-pixel level until the target point is located at the center of the field of view (step 106), and the possible system errors or sample displacements are monitored and corrected in real time throughout the process to ensure the high consistency and accuracy of the positioning results. This method not only enhances the stability of positioning, but also supports the analysis of complex structures, providing a powerful tool for research in fields such as materials science, biology and nanotechnology. By achieving fast and accurate target point positioning, this method greatly improves the work efficiency of microstructure analysis, simplifies the experimental process, and lays a solid foundation for subsequent research and application. In summary, from the initial establishment of the reference marker system to the final sub-pixel level fine-tuning, each step is linked together to form an efficient and accurate scanning electron microscope target point positioning solution, which brings important value to scientific research and technological applications.

[0109] In order to solve the problem that it is difficult to accurately reconstruct the three-dimensional position information of the reference marker in the traditional method and further improve the accuracy and reliability of the target point positioning, in some embodiments, step 104 determines the angle difference and position difference of the same reference marker under different observation angles based on the multiple sample images, and reconstructs the position information of the reference marker in the three-dimensional coordinate system based on the angle difference and position difference through stereo vision technology to form a three-dimensional space geometric relationship network, including:

[0110] For each sample image, a computer vision algorithm is used to extract feature points of the reference marker, and a descriptor is generated for each feature point, wherein the feature point refers to a key point that can uniquely identify the position and shape of the reference marker in the sample image; by comparing the feature descriptors in different sample images, the feature points of the same reference marker from different observation angles are identified, and feature point pairs are generated, wherein the feature point pairs are composed of one or more pairs of feature points from different sample images, and the feature points in each feature point pair have similar feature descriptors and correspond to the same physical position; based on the feature point pairs, geometric constraints are applied to estimate the relative geometric relationship between different observation angles, and the rotation and translation changes of the camera on the extrinsic parameters are calculated by decomposing the relative geometric relationship to obtain the camera attitude information; based on the feature point pairs and the camera attitude information, the relative rotation angle of the same reference marker at different observation angles is calculated as the angle difference, and the relative displacement of the same reference marker at different observation angles is calculated as the position difference; an improved triangulation algorithm is applied, combined with the observation conditions and geometric constraints corresponding to each sample image, and the angle difference and position difference are used to reconstruct the position information of the reference marker in the three-dimensional coordinate system to form a three-dimensional spatial geometric relationship network.

[0111] In this embodiment, the feature points are key points in the sample image that can uniquely identify the position and morphology of the reference markers. These points usually have high contrast or unique geometric characteristics, so that they can be stably identified in images with different observation angles.

[0112] The descriptor is a numerical representation generated for each feature point, which is used to describe the local image information around the point. The descriptor enables feature points at the same physical location to be matched and identified even under different observation angles.

[0113] A feature point pair consists of one or more pairs of feature points from different sample images. The feature points in each feature point pair have similar feature descriptors and correspond to the same physical location. Feature point pairs are the basis for constructing three-dimensional geometric relationships.

[0114] The camera pose information includes the rotation and translation changes of the camera relative to the world coordinate system, i.e., the external parameters. It describes the relative geometric relationship between different observation perspectives and is crucial for reconstructing 3D coordinates.

[0115] Among them, the specific process of "calculating the relative rotation angle of the same reference marker under different observation perspectives and using it as the angle difference, and calculating the relative displacement of the same reference marker under different observation perspectives as the position difference" can be achieved by comparing the angle and straight-line distance difference between the feature point pairs under two observation perspectives.

[0116] Improved triangulation algorithm An optimized algorithm that combines the observation conditions and geometric constraints corresponding to multiple sample images to accurately reconstruct the position of the reference marker in the three-dimensional coordinate system by calculating the angular difference and position difference of the feature point pairs.

[0117] In the embodiment of the present application, a computer vision algorithm is first used to extract feature points of a reference marker from each sample image, and a descriptor is generated for each feature point. Then, by comparing the feature descriptors in different sample images, feature points belonging to the same physical location are identified and feature point pairs are formed. Based on these feature point pairs, geometric constraints are applied to estimate the relative geometric relationship between different observation perspectives, and the camera's posture information (rotation and translation) is calculated.

[0118] Subsequently, based on the feature point pairs and the camera pose information, the relative rotation angle and displacement of the same reference marker under different observation perspectives are calculated as the angle difference and position difference.

[0119] Finally, the improved triangulation algorithm is used in combination with observation conditions and geometric constraints to accurately reconstruct the position information of the reference markers in the three-dimensional coordinate system through the above-mentioned angle differences and position differences, forming a complete three-dimensional spatial geometric relationship network. In this process, possible errors are constantly corrected to ensure the high consistency of the positioning results.

[0120] Here is a specific example:

[0121] In a specific embodiment consistent with the field of artificial intelligence, deep learning technology can be used to enhance the accuracy and efficiency of the above steps. Assume that an automated SEM target point positioning system is being developed. First, a convolutional neural network (CNN) model, such as U-Net or ResNet, is trained specifically for feature point detection and descriptor generation in SEM images. The model is trained with a large amount of annotated SEM image data to learn the best representation of different morphological features, thereby improving the accuracy of feature point matching. Next, a matching network based on an attention mechanism, such as a Transformer, is developed to automatically identify and match feature point pairs under different observation angles. This network can handle complex background interference and further improve matching accuracy.

[0122] In order to estimate the relative geometric relationship between different observation perspectives, a graph neural network (GNN) is used to model the geometric relationship between feature point pairs and predict the camera pose information. GNN can capture the complex interactions between feature point pairs and provide more accurate geometric relationship estimates. Then, an end-to-end deep learning framework such as Pix2Vox or PointNet++ is implemented to reconstruct the 3D model of the reference marker directly from the multi-view images. This method can not only improve the reconstruction speed, but also generate more refined and accurate 3D structures.

[0123] Throughout the process, reinforcement learning algorithms, such as DQN or PPO, are introduced to dynamically adjust the errors that occur during the reconstruction process. The system will continuously optimize the error correction strategy based on the feedback of each adjustment to ensure that the final three-dimensional spatial geometric relationship network is as close to the actual situation as possible. In this way, not only a highly automated process is achieved, but also advanced AI technology is used in all links, which greatly improves the accuracy and efficiency of SEM target point positioning, providing strong support for scientific research and technological applications.

[0124] In order to solve the problem that it is difficult to accurately reconstruct the three-dimensional coordinates and postures of the reference markers in the traditional methods and further improve the accuracy and stability of the target point positioning, in some embodiments, the improved triangulation algorithm is applied, combined with the observation conditions and geometric constraints corresponding to each sample image, and the angle difference and position difference are used to reconstruct the position information of the reference markers in the three-dimensional coordinate system to form a three-dimensional space geometric relationship network, including:

[0125] An improved triangulation algorithm is applied, in combination with the observation conditions and geometric constraints corresponding to each sample image, and the angle difference and the position difference are used to calculate the position information of the feature point pairs in the three-dimensional coordinate system; the three-dimensional coordinates of multiple feature points on each reference marker are obtained by integrating the position information of all feature point pairs; the position information of each reference marker is determined by fitting the three-dimensional coordinates of multiple feature points of each reference marker, and the position information includes three-dimensional coordinates and posture; based on the position information of each reference marker, a three-dimensional spatial geometric relationship network is constructed that describes the relative angles and straight-line distances between all reference markers and between the reference marker and the target point, and in the process of constructing the three-dimensional spatial geometric relationship network, the angle difference and the position difference are used to correct possible errors.

[0126] In this scheme, the position information of each reference marker is determined by fitting the three-dimensional coordinates of multiple feature points of each reference marker. The reason is that since multiple feature points jointly describe the spatial distribution of the reference marker, the three-dimensional coordinates of the multiple feature points can represent the position information of the reference marker in the three-dimensional coordinate system.

[0127] In this embodiment, the improved triangulation algorithm is an optimized geometric reconstruction method that combines observation conditions (such as magnification, viewing angle, etc.) and geometric constraints (such as known camera parameters) corresponding to multiple sample images to accurately reconstruct the position of the reference marker in the three-dimensional coordinate system by calculating the angle difference and position difference of the feature point pair. Compared with traditional triangulation, the improved algorithm can usually better handle noise, reduce error accumulation, and improve reconstruction accuracy.

[0128] Angle difference and position difference refer to the rotation angle change and spatial displacement of the same reference marker under different observation angles. These differences reflect the geometric transformation between different observation angles and are one of the key information for building a 3D model.

[0129] The position information of feature point pairs refers to the specific coordinates of the feature point pairs extracted and successfully matched from multiple sample images in the three-dimensional coordinate system. The feature points in each feature point pair have similar descriptors and correspond to the same physical position. Their position information is used to determine the overall three-dimensional structure of the reference marker.

[0130] The 3D coordinates and pose are the 3D coordinates that represent the exact position of the reference marker in space; the pose describes its orientation relative to the world coordinate system, including rotation and translation. This information is crucial for understanding the spatial distribution of the reference marker and its relative relationship with other objects.

[0131] In the embodiment of the present application, in order to more accurately reconstruct the three-dimensional coordinates and posture of the reference marker, the present solution adopts an improved triangulation algorithm, combines the observation conditions and geometric constraints of each sample image, and uses the angle difference and position difference for calculation. First, the improved triangulation algorithm is applied to calculate the accurate position of each pair of feature points in the three-dimensional coordinate system based on the angle difference and position difference.

[0132] Next, the position information of all feature point pairs is aggregated to obtain the 3D coordinates of multiple feature points on each reference marker, ensuring that the reconstructed 3D model has sufficient details and accuracy. Then, by fitting the 3D coordinates of multiple feature points of each reference marker, the overall position information of the reference marker, including 3D coordinates and posture, is determined. This step helps eliminate the possible errors of a single feature point and provides a more stable positioning benchmark.

[0133] Finally, based on the position information of each reference marker, a complete three-dimensional spatial geometric relationship network is constructed to describe the relative angles and straight-line distances between all reference markers and between the reference markers and the target point. In this process, the angle differences and position differences are used to correct possible errors to ensure the high consistency and reliability of the entire network.

[0134] Here is a specific example:

[0135] Assuming that you are developing an automated SEM target point positioning system, you can use the following techniques to enhance the accuracy and efficiency of the above steps. Train a convolutional neural network (CNN), such as ResNet or EfficientNet, specifically for feature point detection and descriptor generation in SEM images. The model can be trained with a large amount of annotated SEM image data to learn the best representation of different morphological features, thereby improving the accuracy of feature point matching. For example, use a pre-trained ResNet model as a basis and add a fine-tuning layer specific to SEM images to adapt to different material and structural properties.

[0136] Develop an attention-based matching network, such as Transformer, to automatically identify and match feature point pairs from different observation perspectives. This network can handle complex background interference and further improve matching accuracy. For example, use the self-attention mechanism in the Transformer model to enhance the correlation between feature point pairs, so as to more reliably identify the same feature points from different observation perspectives.

[0137] Graph Neural Networks (GNN) are used to model the geometric relationship between feature point pairs and predict camera pose information. GNN can capture the complex interactions between feature point pairs and provide more accurate geometric relationship estimates. For example, using GNN models such as GraphSAGE or GAT (Graph Attention Networks), a graph structure is constructed based on feature point pairs, and node features are updated through a message passing mechanism to ultimately obtain the camera's external parameters (rotation and translation).

[0138] Implement an end-to-end deep learning framework, such as Pix2Vox or PointNet++, to reconstruct the 3D model of the reference marker directly from the multi-view images. This framework can not only improve the reconstruction speed, but also generate more refined and accurate 3D structures. At the same time, in the process of building the 3D spatial geometric relationship network, a reinforcement learning algorithm (such as DQN or PPO) is introduced to dynamically adjust the errors that occur during the reconstruction process. The system will continuously optimize the error correction strategy based on the feedback of the effect after each adjustment to ensure that the final 3D spatial geometric relationship network is as close to the actual situation as possible.

[0139] In order to solve the problem that it is difficult to accurately restore the spatial geometric relationship between the target point and the reference markers in the traditional method, and further improve the accuracy and efficiency of the preliminary positioning of the target point, in some embodiments, the step 105 uses the three-dimensional spatial geometric relationship network to obtain the position information of the reference markers at the current level and the relative angles and straight-line distances between the reference markers, and calculates the preliminary position of the target point, including:

[0140] The three-dimensional spatial geometric relationship network is used to obtain the position information of the reference markers and the relative angles between the reference markers under the current observation conditions to reproduce the observation conditions when the sample to be tested is first observed under the scanning electron microscope; the spatial geometric relationship between the target point and the reference markers is restored based on the relative angle and straight-line distance of the target point recorded under the observation conditions when the sample to be tested is first observed under the scanning electron microscope, combined with the position information of the reference markers and the relative angles and straight-line distances between the reference markers under the current observation conditions; the preliminary position of the target point is calculated using the spatial geometric relationship through an improved triangulation algorithm.

[0141] In this embodiment,

[0142] Observation conditions refer to the specific parameters recorded during the first SEM observation, such as magnification, viewing angle, sample stage angle, etc. These conditions are crucial to reproducing the relative position of the target point during the first observation.

[0143] The spatial geometric relationship refers to the relative position relationship between the target point and the reference marker, including the relative angle and straight-line distance. Restoring this relationship is the basis for calculating the preliminary position of the target point.

[0144] The improved triangulation algorithm is an optimized geometric reconstruction method that combines the current observation conditions and the geometric constraints in the historical observation data to accurately reconstruct the preliminary position of the target point by calculating the angle difference and position difference of the feature point pair. Compared with the traditional method, the improved algorithm can better handle noise, reduce error accumulation, and improve positioning accuracy.

[0145] In the embodiment of the present application, the three-dimensional spatial geometric relationship network is used to obtain the position information of the reference markers and the relative angles between the reference markers under the current observation conditions. First, the observation conditions when the sample to be tested is first observed by the scanning electron microscope are reproduced through the network.

[0146] Then, based on the relative angle and straight-line distance of the target point relative to the reference marker recorded under the first observation conditions, combined with the position information of the reference markers under the current observation conditions and the relative angle and straight-line distance between the reference markers, the spatial geometric relationship between the target point and the reference markers is restored.

[0147] Finally, using this spatial geometric relationship, the preliminary position of the target point is calculated through an improved triangulation algorithm. This process ensures that the approximate position of the target point can be accurately found even under different observation conditions, providing a reliable basis for further precise positioning.

[0148] Here is a specific example:

[0149] Assuming that an automated SEM target point relocation system is being developed, the following techniques can be used to enhance the accuracy and efficiency of the above steps:

[0150] Use a deep learning model (such as LSTM or Transformer) to analyze the observation conditions (such as magnification, viewing angle, sample stage angle, etc.) at the time of the first observation, and predict how to adjust the current observation conditions to be closest to the state of the first observation. This step helps to minimize the impact of changes in observation conditions.

[0151] Based on the 3D spatial geometric relationship network, combined with the relative angle and straight-line distance of the target point relative to the reference marker recorded during the first observation, a graph neural network (GNN) model, such as GraphSAGE or GAT, is developed to automatically match and restore the spatial geometric relationship between the target point and the reference marker. GNN can capture the complex interactions between feature point pairs and provide more accurate spatial relationship estimates.

[0152] Implement an end-to-end deep learning framework, such as Pix2Vox or PointNet++, to reconstruct the 3D model of the reference marker directly from the multi-view images. On this basis, introduce an improved triangulation algorithm to accurately calculate the preliminary position of the target point by combining the current observation conditions and the geometric constraints in the historical observation data. For example, use the pre-trained PointNet++ model as a basis and add a fine-tuning layer specific to SEM images to adapt to different material and structural properties, thereby improving the positioning accuracy.

[0153] Throughout the process, reinforcement learning algorithms (such as DQN or PPO) are introduced to dynamically adjust the errors that occur during the reconstruction process. The system will continuously optimize the error correction strategy based on the feedback of each adjustment to ensure that the final three-dimensional spatial geometric relationship is as close to the actual situation as possible. In addition, self-supervised learning methods can be used to further improve the generalization ability and robustness of the model through contrastive learning or consistency regularization.

[0154] In order to solve the problem that it is difficult to accurately reproduce the first observation conditions in the traditional method and further improve the accuracy and consistency of the target point relocation, in some embodiments, the acquisition of the position information of the reference markers and the relative angles between the reference markers under the current observation conditions to reproduce the observation conditions when the sample to be tested is first observed by the scanning electron microscope includes:

[0155] Based on the three-dimensional spatial geometric relationship network, the three-dimensional coordinates of each reference marker under the current observation conditions are obtained as the position information of the reference marker; based on the position information, the relative angles between all reference markers in the same layer are calculated to determine the relative angles between the reference markers under the current observation conditions; the relative angles and straight-line distances between the reference markers under the current observation conditions, as well as the angle of the sample stage, are adjusted to ensure that the relative angles and straight-line distances between the reference markers are consistent with the relative angles and straight-line distances originally determined between the reference markers during the first observation, and that the angle of the sample stage is consistent with the angle of the sample stage originally determined.

[0156] In this embodiment, the relative angle and linear distance refer to the geometric relationship between reference markers in the same layer, that is, the angle and linear distance between them. Ensuring the consistency of these parameters is the basis for accurate repositioning.

[0157] The sample stage angle adjustment refers to adjusting the angle of the platform on which the sample is placed relative to the optical axis of the SEM to restore the observation conditions at the time of the first observation. This step ensures geometric consistency under different observation conditions.

[0158] In an embodiment of the present application, in order to more accurately reproduce the initial observation conditions and further improve the accuracy and consistency of target point repositioning, this scheme uses a three-dimensional spatial geometric relationship network to obtain the three-dimensional coordinates of each reference marker under the current observation conditions as its position information.

[0159] Then, the relative angles between all reference markers in the same layer are calculated based on this position information to determine the relative angles between the reference markers under the current observation conditions. Next, the relative angles and linear distances between the reference markers under the current observation conditions are adjusted, and the angle of the sample stage is adjusted at the same time to ensure that these geometric relationships remain consistent with the initial observation. This process not only ensures the consistency of the relative angles and linear distances between the reference markers, but also ensures that the angle of the sample stage is the same as the angle at the initial observation, thus providing a reliable benchmark for subsequent precise positioning.

[0160] Here is a specific example:

[0161] Assuming that an automated SEM target point relocation system is being developed, the following techniques can be used to enhance the accuracy and efficiency of the above steps:

[0162] Use a deep learning model (such as PointNet++ or Pix2Vox) to extract and reconstruct the 3D coordinates of each reference marker from the multi-view images under the current observation conditions. The model can be trained with a large amount of annotated SEM image data to learn the best representation of different morphological features, thereby improving the accuracy of 3D coordinate extraction.

[0163] Based on the extracted 3D coordinates, a graph neural network (GNN) model, such as GraphSAGE or GAT, is developed to automatically calculate the relative angles between all reference markers in the same hierarchy. GNN is able to capture the complex interactions between feature point pairs and provide more accurate spatial relationship estimates.

[0164] Introduce reinforcement learning algorithms (such as DQN or PPO) to dynamically adjust the relative angles and straight-line distances between reference markers, as well as the angle of the sample stage. The system will continuously optimize the adjustment strategy based on the feedback of each adjustment to ensure that the final geometric relationship is as close as possible to the state at the time of the first observation. For example, use self-supervised learning methods, through contrastive learning or consistency regularization, to further improve the generalization ability and robustness of the model.

[0165] During the whole process, the adjusted geometric relationship and sample stage angle are monitored and recorded in real time to ensure that they are highly consistent with the data at the time of the first observation. If any deviation is found, it is corrected immediately to maintain high-precision repositioning. In addition, a closed-loop control system can be introduced to achieve real-time error correction to ensure the stability and reliability of the entire repositioning process.

[0166] In order to solve the problem of inaccurate target point positioning caused by changes in sample microstructure in traditional methods and further improve the accuracy and stability of target point repositioning, some embodiments further include:

[0167] When the sample to be tested is observed by scanning electron microscopy for the first time, the multimodal data of the target point and the surrounding area are recorded and stored, and the multimodal data at least include spectral features and physical properties; when the sample to be tested is processed and observed by scanning electron microscopy for the second time, the multimodal data of the current target point and the surrounding area are re-collected; by comparing the multimodal data stored during the first observation with the multimodal data currently collected, a multimodal feature matching result is generated, and the multimodal feature matching result includes change information of the target point and the surrounding area between the two observations; based on the multimodal feature matching result, the target position of the target point is adjusted to ensure the positioning accuracy of the target point, and it is not affected by the microstructural changes of the sample to be tested.

[0168] In this embodiment, multimodal data refers to various types of data of the target point and the surrounding area recorded and stored during the first SEM observation, including at least spectral features (such as energy dispersive X-ray spectroscopy EDS, Raman spectroscopy, etc.) and physical properties (such as surface morphology, texture, roughness, etc.). These data provide comprehensive information about the target point and its environment for subsequent feature matching and positioning adjustment.

[0169] Spectral features refer to sample composition or structural information obtained through spectral analysis techniques, such as energy dispersive X-ray spectroscopy (EDS) can reveal the elemental composition of the sample, while Raman spectroscopy can provide information on the molecular structure. These features are very important for identifying and comparing the same location under different observation conditions.

[0170] Physical refers specifically to the morphological characteristics of the sample surface, such as surface topography, texture, roughness, etc., which are usually obtained through scanning electron microscope images or other microscopic imaging techniques. These characteristics help to accurately describe the microstructure of the sample and provide more details for feature matching.

[0171] The multimodal feature matching result is generated by comparing the multimodal data stored at the time of the first observation with the multimodal data currently collected, and contains information about the changes in the target point and the surrounding area between the two observations. This matching result not only reveals the changes in the microstructure, but also provides the reference information needed to adjust the position of the target point.

[0172] In the embodiment of the present application, in order to more accurately deal with the influence of sample microstructure changes on the positioning of the target point and further improve the accuracy and stability of repositioning, this solution records and stores multimodal data of the target point and the surrounding area when the sample to be tested is first observed by a scanning electron microscope, and these data include at least spectral characteristics and physical properties. When the sample to be tested is subjected to a second scanning electron microscope observation after being processed, the multimodal data of the current target point and the surrounding area are re-collected.

[0173] Then, by comparing the multimodal data stored during the first observation with the currently collected data, a multimodal feature matching result is generated, which contains the change information of the target point and the surrounding area between the two observations.

[0174] Finally, based on these multimodal feature matching results, the target position of the target point is adjusted to ensure that its positioning accuracy is not affected by changes in the sample's microstructure. This process ensures that high-precision target point repositioning can be achieved even if the sample surface features change.

[0175] Here is a specific example:

[0176] Assuming that an automated SEM target point relocation system is being developed, the following techniques can be used to enhance the accuracy and efficiency of the above steps:

[0177] During the first SEM observation, an integrated multi-sensor platform (such as SEM combined with EDS and Raman spectrometer) is used to synchronously collect multimodal data of the target point and the surrounding area. These data will be stored in an efficient data management system to support rapid retrieval and comparison. For example, a database management system (DBMS) combined with a cloud storage service is used to ensure data security and accessibility.

[0178] When the sample is processed and then observed for the second time using the SEM, the same multi-sensor platform is used again to collect multimodal data of the target point and surrounding area. In order to ensure data consistency, standardized operating procedures and parameter configurations can be set in the system to ensure that the two observation conditions are as consistent as possible.

[0179] Develop a deep learning model, such as a convolutional neural network (CNN) combined with a recurrent neural network (RNN), to automatically compare the multimodal data stored at the time of the first observation with the currently collected data. The model can be trained with a large amount of annotated multimodal data to learn the correlation between different features, thereby generating accurate multimodal feature matching results. For example, using ResNet combined with an LSTM model can effectively capture feature changes in space and time.

[0180] Based on the generated multimodal feature matching results, a reinforcement learning algorithm (such as DQN or PPO) is introduced to dynamically adjust the position of the target point. The system will continuously optimize the adjustment strategy based on the feedback of the effect after each adjustment to ensure that the final positioning result is as close to the actual situation as possible. In addition, a closed-loop control system can be introduced to achieve real-time error correction to ensure the stability and reliability of the entire relocation process.

[0181] During the whole process, the adjusted geometric relationship and sample stage angle are monitored and recorded in real time to ensure that they are highly consistent with the data at the time of the first observation. If any deviation is found, it is corrected immediately to maintain high-precision repositioning. In addition, self-supervised learning methods can be introduced to further improve the generalization ability and robustness of the model through contrastive learning or consistency regularization.

[0182] This application considers that traditional triangulation algorithms often ignore the influence of observation conditions and geometric constraints (such as basic matrix) on the accuracy of feature point matching when processing multi-view image reconstruction. In addition, traditional weight calculation methods fail to fully consider the influence of factors such as angle difference, position difference and camera translation vector on three-dimensional coordinate reconstruction, resulting in insufficient accuracy of reconstructed three-dimensional coordinates under complex backgrounds or with large changes in perspective. Therefore, a new optional solution is proposed, which includes:

[0183] The geometric constraint includes a basic matrix; the application of the improved triangulation algorithm, combining the observation conditions and geometric constraints corresponding to each sample image, and using the angle difference and the position difference to calculate the position information of the feature point pair in the three-dimensional coordinate system, includes:

[0184] The position information of the feature point pair in the three-dimensional coordinate system is calculated by the following calculation formula, wherein the position information includes the three-dimensional coordinates;

[0185]

[0186] w′ i =w i β i

[0187] w i =exp(-α·||T i || 2 -β·||Δp i || 2 )

[0188] β i =exp(γ·θ i )

[0189] Among them, P is the three-dimensional coordinate of the feature point, n is the number of feature point pairs, i is the index of the feature point pair, and w' i is the comprehensive weight of the i-th group of feature point pairs, K i is the camera intrinsic parameter matrix corresponding to the i-th set of feature point pairs, x 1,i and x 2,i are the 2D coordinates of the same reference marker in two different sample images, and F i is the basic matrix corresponding to the i-th group of feature point pairs, wi is the initial weight of the i-th group of feature point pairs, β i is the exponential adjustment factor of the i-th group of feature point pairs, θ is the angle difference, Δp is the position difference, α is a positive adjustment parameter used to control the influence of the translation vector, T i is the translation vector corresponding to the i-th set of feature point pairs, ||T i || represents the Euclidean norm of the translation vector, β is a positive adjustment parameter used to control the influence of position difference, Δp i is the position difference of the i-th group of feature point pairs, indicating the relative displacement between two different observation perspectives, ||Δp i || 2 is the Euclidean norm of the position difference, γ is a positive adjustment parameter used to control the influence of the rotation angle, θ i is the angle difference of the i-th set of feature point pairs, indicating the relative rotation angle between two different observation perspectives.

[0190] The following is a detailed explanation of each parameter:

[0191] P: The specific position of the feature point in the three-dimensional coordinate system obtained by calculation. It is the final reconstruction target and represents the precise position information of the reference marker in space. This value is calculated by weighted average in the formula, which ensures geometric consistency under different observation conditions and improves positioning accuracy.

[0192] n: The total number of feature point pairs used for reconstruction. Each feature point pair consists of two feature points at the same physical location in images from different observation angles. This number determines the richness of the dataset. More feature point pairs usually provide higher reconstruction accuracy and stability.

[0193] i: An index variable of feature point pairs, used to identify each pair of feature points. It goes from 1 to n, traversing all feature point pairs involved in the calculation. This index helps process the data of each feature point pair one by one, ensuring that each matching pair is appropriately weighted and considered.

[0194] w' i : The comprehensive weight of the i-th group of feature point pairs, combined with the initial weight w i and exponential adjustment factor β i The comprehensive weight reflects the importance of each feature point pair to the final reconstruction result, ensuring that more reliable and stable feature point pairs have a larger proportion in the calculation. The calculation formula is: w' i =w i β i

[0195] w i : The initial weight of the i-th group of feature point pairs, based on the translation vector T i and position difference Δp i The calculation formula of initial weight is: i =exp(-α·||T i || 2 -β·||Δp i || 2 )

[0196] Among them, α and β are positive adjustment parameters used to control the influence of translation vector and position difference on weight. Larger α or β values ​​will reduce the weight of feature point pairs with large translation or position difference, thereby reducing their influence on the final result.

[0197] β i : The exponential adjustment factor of the i-th group of feature point pairs, mainly reflecting the angle difference θ i The impact of feature points on reliability.

[0198] The calculation formula is: i =exp(γ·θ i )

[0199] Among them, γ is a positive adjustment parameter used to control the degree of influence of the rotation angle. Smaller angle differences will lead to β i is close to 1, while a larger angle difference will make β i Significantly increase or decrease, thereby adjusting the comprehensive weight of the feature point pair.

[0200] K i : The camera intrinsic parameter matrix corresponding to the i-th set of feature point pairs contains the camera's intrinsic parameters such as focal length and principal point position. These parameters are crucial for converting two-dimensional image coordinates into three-dimensional world coordinates. In the formula, is used to back-project image coordinates into three-dimensional space, and It is used to process the basic matrix F i transformation.

[0201] x 1,i and x 2,i : The two-dimensional coordinates of the same reference marker in two different sample images. These two coordinates are obtained through feature point detection and matching algorithms to describe the same physical location under different observation angles. They are the basis for building geometric relationships and the starting point for calculating three-dimensional coordinates.

[0202] F i : The basic matrix corresponding to the i-th set of feature point pairs describes the geometric relationship between the two images. The basic matrix contains information about the camera motion, including rotation and translation, and is used to associate the coordinates of the same physical location under different observation angles. In the formula, F i It is used to calculate the relative position relationship between feature point pairs.

[0203] θ i : The relative rotation angle of the i-th set of feature point pairs between two different observation perspectives. The angle difference reflects the rotation change between the two observation perspectives, which is very important for evaluating the reliability of feature point pairs and adjusting the comprehensive weights. Larger angle differences may lead to larger errors, so it is necessary to adjust the angle by β i Make appropriate adjustments.

[0204] Δp i : The relative displacement of the i-th set of feature point pairs between two different observation perspectives. The position difference reflects the actual moving distance of the feature points between the two observation perspectives, which is very important for evaluating the stability of the feature point pairs and adjusting the initial weights. A large position difference may reduce the reliability of the feature point pairs, so it is necessary to use w i Make appropriate adjustments.

[0205] α, β, and γ: positive tuning parameters, α controls the translation vector T i For the initial weight w i The degree of influence of β controls the position difference Δp i For the initial weight w i The degree of influence. γ controls the rotation angle θ i Exponential adjustment factor β i degree of impact.

[0206] Here is a specific example:

[0207] Suppose you are developing an automated SEM target point relocation system and want to use the above formula to improve the accuracy of the feature point 3D coordinate calculation. You will use the Python programming language and the OpenCV library to implement it and substitute specific values ​​for the calculation.

[0208] Parameter settings:

[0209] n: The number of feature point pairs, set to 5.

[0210] The camera intrinsic parameter matrix, assuming that all camera intrinsic parameters are the same, is simplified to the identity matrix 1.

[0211] x 1,i and x 2,i : The two-dimensional coordinates of the same reference marker in two different sample images (see Table 1).

[0212] F i : The basic matrix, assumed to be a known value, is simplified to the identity matrix I.

[0213] α, β, γ: adjustment parameters, set to 0.5, 0.3, 0.2 respectively.

[0214] θ i : Angle difference, assuming a small angle change, for example θ1 = 0.1 radian.

[0215] Δp i : Position difference, assumed to be a small displacement change, for example Δp1 = [2, -3] pixels.

[0216] T i : Translation vector, assuming a small translation change, for example T1 = [1,2,3].

[0217] An example of feature point pair coordinates and parameters is shown in Table 1 below:

[0218] Table 1

[0219] i <![CDATA[x 1,i ]]> <![CDATA[x 2,i ]]> <![CDATA[T i ]]> <![CDATA[Δp i ]]> <![CDATA[θ i ]]> 1 [100,150] [105,148] [1,2,3] [2,-3] 0.1 2 [200,300] [203,297] [1,2,3] [3,-3] 0.15 3 [300,450] [305,448] [1,2,3] [5,-2] 0.2 4 [400,600] [403,597] [1,2,3] [3,-5] 0.1 5 [500,750] [505,748] [1,2,3] [5,-3] 0.15

[0220] Calculation process:

[0221] For the first set of feature point pairs:

[0222] w1=exp(-0.5·(1 2 +2 2 +3 2 )-0.3·(2 2 +(-3) 2 )) = exp(-0.5·14-0.3·13)

[0223] ≈0.0006

[0224] β1=exp(0.2·0.1)≈1.02

[0225] w′1=0.0006·1.02≈0.000612

[0226] Similarly, calculate the comprehensive weight w' of other feature point pairs i , as shown in Table 2 below:

[0227] Table 2

[0228] i 1 2 3 4 5 <![CDATA[w i ]]> 0.000612 0.000558 0.000564 0.000588 0.000564

[0229] For the first set of feature point pairs:

[0230]

[0231] Summarize and calculate the final 3D coordinate P:

[0232] Molecular part:

[0233]

[0234] Denominator:

[0235]

[0236] Final three-dimensional coordinates P:

[0237]

[0238] P≈0.00173i+0.001983j-0.00833k

[0239] Through the above calculations, the final three-dimensional coordinates P≈(0.00173, 0.001983, -0.00833) are obtained. These calculation results show that even in the presence of noise, translation and rotation, the improved triangulation algorithm can still effectively use the feature point pairs in the multi-view image to generate more accurate three-dimensional coordinates. In particular, by introducing comprehensive weights and exponential adjustment factors, the emphasis on key feature points is enhanced, the influence of irrelevant feature points is reduced, and the robustness and accuracy of three-dimensional reconstruction are improved.

[0240] This application takes into account that when dealing with target point relocation, traditional methods are difficult to accurately restore the spatial geometric relationship under the first observation conditions, especially when the sample undergoes microstructural changes after processing. This leads to inaccurate initial position calculations of the target points, affecting the reliability of subsequent precise positioning and analysis. In order to meet this challenge and further improve the accuracy and stability of target point relocation, a new optional solution is proposed, which includes:

[0241] The method of using the three-dimensional spatial geometric relationship network to obtain the position information of the reference markers at the current level and the relative angles and straight-line distances between the reference markers, and calculating the preliminary position of the target point, includes:

[0242] Determine the position information of each reference marker in the three-dimensional coordinate system under the current observation conditions

[0243] Calculate relative angles between reference markers and straight-line distance

[0244] The linear distance d between the target point and the reference marker recorded under the observation conditions when the sample to be tested is first observed by the scanning electron microscope initial,ref , combined with the relative angles between the reference markers under the current observation conditions and straight-line distance Restore the spatial geometric relationship between the target point and the reference marker, and calculate the preliminary position P of the target point by the following formula initial :

[0245]

[0246] Where n is the number of reference markers; i is the index of the reference marker; is the weight of the i-th reference marker, according to the distance d from the target point to the reference marker under the first observation condition initial,ref The straight-line distance between the reference marker under the current observation conditions Determined, represents the straight-line distance between the i-th reference marker and all other reference markers j The sum of is the position information of the i-th reference marker in the three-dimensional coordinate system under the current observation conditions; ΔP i is the displacement vector of the target point relative to the i-th reference marker derived based on the spatial geometric relationship between the first observation condition and the current observation condition, and the displacement vector is calculated by the following formula:

[0247]

[0248] where θ initial,ij is the relative angle between the i-th reference marker and the j-th reference marker under the first observation condition; is the relative angle between the i-th reference marker and the j-th reference marker under the current observation condition; is the unit vector pointing from the i-th reference marker to the j-th reference marker.

[0249] The following is a detailed explanation of each parameter:

[0250] P initial : The initial position of the target point, which indicates the position of the target point relative to the reference marker restored by calculation under the current observation conditions. This position is based on the spatial geometric relationship under the first observation conditions and the reference marker position information under the current observation conditions. It is a three-dimensional coordinate vector in the form of [x, y, z], where x, y, and z represent the horizontal coordinate, vertical coordinate, and height coordinate of the target point in the three-dimensional coordinate system, respectively.

[0251] n: The number of reference markers. Reference markers are key points that can uniquely identify the position and morphology in the SEM image and are used to construct a three-dimensional spatial geometric relationship network. These markers provide important information about the sample structure and serve as the basis for calculating the preliminary position of the target point. The value of n depends on the number of reference markers selected in the actual experiment. Generally, more reference markers can provide more accurate spatial geometric relationships.

[0252] w i : The weight of the i-th reference marker, which is used to measure the influence of the reference marker on the preliminary position calculation of the target point. The weight calculation formula is:

[0253] Here, d initial,ref is the distance from the target point to the i-th reference marker under the first observation condition, and is the sum of the straight-line distances between the i-th reference marker and all other reference markers. The weight reflects the importance of the distance between the target point and each reference marker when first observed. The closer the reference marker, the greater the impact on the final result.

[0254] is the position information of the i-th reference marker in the three-dimensional coordinate system under the current observation conditions. It is a three-dimensional coordinate vector in the form of [x i ,y i ,z i ] represents the specific position of the i-th reference marker under the current observation condition. This position information is obtained through a three-dimensional spatial geometric relationship network, ensuring geometric consistency under different observation conditions.

[0255] ΔP i is the displacement vector of the target point relative to the i-th reference marker, indicating the change in direction and distance from the i-th reference marker to the target point. It is derived from the spatial geometric relationship between the first observation condition and the current observation condition, and is used to correct the target point position offset caused by changes in the sample microstructure.

[0256] d initial,ref,i is the straight-line distance from the target point to the i-th reference marker under the first observation condition. This distance is obtained by recording the data during the first SEM observation and reflects the relative position relationship between the target point and the reference marker. It is used in calculating the weight w i and displacement vector ΔP i It plays a key role in ensuring the consistency of target point positioning even under different observation conditions.

[0257] is the relative angle between the i-th reference marker and the j-th reference marker under the current observation conditions. This angle describes the directional relationship between the two reference markers and is obtained through multi-view image analysis under the current observation conditions. Changes in the relative angle reflect changes in the sample's microstructure and are essential for adjusting the position of the target point.

[0258] θ initial,ij is the relative angle between the i-th reference marker and the j-th reference marker under the first observation condition. This angle is obtained from the data recorded during the first SEM observation and reflects the directional relationship between the two reference markers at the time of the first observation. It is different from the relative angle under the current observation condition. Comparison is performed to evaluate changes in the sample microstructure and to calculate the displacement vector ΔP i .

[0259] is the unit vector pointing from the i-th reference marker to the j-th reference marker. The unit vector Describes the directional relationship between the two reference markers and has a modulus of 1. The unit vector is used in calculating the displacement vector ΔP i is used to determine the direction, ensuring that the direction of the displacement vector correctly reflects the relative position change between the reference markers.

[0260] is a unit vector The modulus of is its length. is a unit vector, so its magnitude is always 1. In the formula, It is mainly used for normalization operations to ensure that the contribution ratio of each vector is consistent during the calculation process. It is always 1, but it is retained in the formula to maintain the integrity of the mathematical expression and facilitate understanding and expansion.

[0261] Here is a specific example:

[0262] Suppose you are developing an automated SEM target point relocation system and want to use the above formula to improve the accuracy of the target point preliminary position calculation. You will use the Python programming language and the NumPy library to implement it and substitute specific values ​​for the calculation.

[0263] Parameter settings:

[0264] n: number of reference markers, set to 3;

[0265]

[0266] d initial,ref : The straight-line distance of the target point relative to each reference marker under the first observation condition, set to 100 unit length;

[0267] and

[0268]

[0269] θ inititul,ij :

[0270]

[0271]

[0272] Calculation process:

[0273] Calculate the weight w i :

[0274]

[0275] Calculate the displacement vector ΔP i :

[0276] For i=1:

[0277]

[0278] Similarly, calculate other ΔP i , since the angle difference is zero, all ΔP i All are zero;

[0279] Calculate the initial position P of the target point initial :

[0280]

[0281] P initial =0.107·(100,150,200)+0.136·(200,300,400)+0.098(300,450,600)

[0282] P initial =(0.107 100 + 0.136 200 + 0.098 300, 0.107 150 + 0.136 300 + 0.098 450, 0.107 200 + 0.136 400 + 0.098 600)

[0283] P initial =(10.7+27.2+29.4,16.05+40.8+44.1,21.4+54.4+58.8)

[0284] P initial =(67.3,100.95,134.6)

[0285] Through the above calculation, the initial position P of the target point is obtained. initial ≈(67.3,100.95,134.6); These calculation results show that even in the presence of noise, translation and rotation, the improved triangulation algorithm can still effectively use the feature point pairs in multi-view images to generate more accurate three-dimensional coordinates. In particular, by introducing comprehensive weights and exponential adjustment factors, the emphasis on key feature points is enhanced, the influence of irrelevant feature points is reduced, and the robustness and accuracy of three-dimensional reconstruction are improved.

[0286] Figure 2 A schematic diagram of a device (or system) for quickly locating a target point based on scanning electron microscope observation is provided for the present application embodiment. Figure 2 As shown, the device comprises:

[0287] A construction module 21 is used to select a target point on the sample to be tested when the sample to be tested is observed by a scanning electron microscope for the first time, and to establish a multi-level, multi-scale reference marker system around the target point, and each level of the reference marker system contains at least three non-collinear morphological features as reference markers, so as to ensure that the target point selected by the sample to be tested at different magnifications has a corresponding reference marker system as a positioning reference;

[0288] An acquisition module 22 is used to adjust the angle of the sample stage on which the sample to be tested is placed, and each time the angle is adjusted and a sample image including the target point and the reference markers around the target point is acquired through a scanning electron microscope, simultaneously record the observation conditions when the sample image is acquired, including the angle of the sample stage and the magnification of the scanning electron microscope;

[0289] A determination module 23, for determining, from each sample image, the relative angle and linear distance between the target point and the reference markers selected at each level and between the reference markers;

[0290] A generating module 24 is used to determine the angle difference and position difference of the same reference marker under different observation angles based on the multiple sample images, and reconstruct the position information of the reference marker in the three-dimensional coordinate system based on the angle difference and position difference through stereo vision technology to form a three-dimensional space geometric relationship network;

[0291] The calculation module 25 is used to determine the reference markers of the corresponding layer according to the current magnification when the sample to be tested is processed and subjected to a second scanning electron microscope observation, obtain the position information of the reference markers of the current layer and the relative angles between the reference markers by using the three-dimensional spatial geometric relationship network, and calculate the preliminary position of the target point;

[0292] The determination module 23 is also used to apply a local fine-tuning algorithm, combined with the sample image collected in real time, and move the sample stage at a sub-pixel level according to the preliminary position of the target point until the target point is located at the center of the field of view, so as to determine the target position of the target point, and during the movement of the sample stage, by monitoring and correcting possible system errors or sample displacements, ensure that the originally determined relative angles and straight-line distances between the reference markers remain consistent.

[0293] Figure 2 The target point rapid positioning device based on scanning electron microscope observation can be performed Figure 1 The implementation principle and technical effect of the target point rapid positioning method based on scanning electron microscope observation described in the illustrated embodiment will not be repeated. The specific manner in which each module and unit performs operations in the target point rapid positioning device based on scanning electron microscope observation in the above embodiment has been described in detail in the embodiment of the method, and will not be elaborated here.

[0294] In one possible design, Figure 2 The target point rapid positioning device based on scanning electron microscope observation in the embodiment shown can be implemented as a computing device, such as Figure 3 As shown, the computing device may include a storage component 31 and a processing component 32;

[0295] The storage component 31 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component 32 .

[0296] The processing component 32 is used for the above Figure 1The embodiment provides a method for quickly locating a target point based on scanning electron microscope observation.

[0297] The processing component 32 may include one or more processors to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing component may also be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0298] The storage component 31 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0299] Of course, the computing device may also include other components, such as input / output interfaces, display components, communication components, etc.

[0300] The input / output interface provides an interface between the processing component and the peripheral interface module, which may be an output device, an input device, etc.

[0301] The communication component is configured to facilitate, among other things, wired or wireless communications between the computing device and other devices.

[0302] Among them, the computing device can be a physical device or an elastic computing host provided by a cloud computing platform, etc. In this case, the computing device can refer to a cloud server, and the above-mentioned processing components, storage components, etc. can be basic server resources rented or purchased from the cloud computing platform.

[0303] The present application also provides a computer storage medium storing a computer program, wherein the computer program can achieve the above-mentioned Figure 1 The illustrated embodiment is a method for quickly locating a target point based on scanning electron microscope observation.

[0304] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0305] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0306] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0307] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for quickly locating a target point based on scanning electron microscope observation, characterized in that: include: When the sample to be tested is observed by scanning electron microscopy for the first time, a target point is selected on the sample to be tested, and a multi-level, multi-scale reference marker system is established around the target point, and each level of the reference marker system contains at least three non-collinear morphological features as reference markers, so as to ensure that the target point selected by the sample to be tested at different magnifications has a corresponding reference marker system as a positioning reference; By adjusting the angle of the sample stage on which the sample to be tested is placed, each time the angle is adjusted and a sample image including the target point and the reference markers around the target point is acquired through a scanning electron microscope, the observation conditions when the sample image is acquired are simultaneously recorded, including the observation angle, the sample stage angle, and the magnification of the scanning electron microscope; Determining from each sample image the relative angle and linear distance between the target point and the reference markers selected at each level and between the reference markers; Based on the multiple sample images, determine the angle difference and position difference of the same reference marker under different observation angles, and reconstruct the position information of the reference marker in a three-dimensional coordinate system based on the angle difference and position difference through stereo vision technology to form a three-dimensional spatial geometric relationship network; When the sample to be tested is processed and subjected to a second scanning electron microscope observation, the reference markers of the corresponding level are determined according to the current magnification, and the position information of the reference markers of the current level and the relative angles and straight-line distances between the reference markers are obtained by using the three-dimensional spatial geometric relationship network, and the preliminary position of the target point is calculated; A local fine-tuning algorithm is applied in combination with the sample image collected in real time. According to the preliminary position of the target point, the sample stage is moved at the sub-pixel level until the target point is located at the center of the field of view, so as to determine the target position of the target point. During the movement of the sample stage, possible system errors or sample displacements are monitored and corrected to ensure that the originally determined relative angles and straight-line distances between the reference markers remain consistent.

2. The method according to claim 1, characterized in that The method of determining the angle difference and position difference of the same reference marker under different observation angles based on the multiple sample images, and reconstructing the position information of the reference marker in a three-dimensional coordinate system based on the angle difference and position difference through stereo vision technology to form a three-dimensional spatial geometric relationship network includes: For each sample image, a computer vision algorithm is used to extract feature points of the reference marker, and a descriptor is generated for each feature point, wherein the feature point refers to a key point that can uniquely identify the position and shape of the reference marker in the sample image; By comparing feature descriptors in different sample images, feature points of the same reference marker from different observation angles are identified, and feature point pairs are generated, wherein the feature point pairs are composed of one or more pairs of feature points from different sample images, and the feature points in each feature point pair have similar feature descriptors and correspond to the same physical position; Based on the feature point pairs, geometric constraints are applied to estimate the relative geometric relationship between different observation perspectives, and the rotation and translation changes of the camera on the extrinsic parameters are calculated by decomposing the relative geometric relationship to obtain the camera posture information; According to the feature point pairs and the camera posture information, the relative rotation angle of the same reference marker at different observation angles is calculated as the angle difference, and the relative displacement of the same reference marker at different observation angles is calculated as the position difference; An improved triangulation algorithm is applied, combined with the observation conditions and geometric constraints corresponding to each sample image, and the angle difference and position difference are used to reconstruct the position information of the reference marker in the three-dimensional coordinate system to form a three-dimensional spatial geometric relationship network.

3. The method according to claim 2, characterized in that The improved triangulation algorithm is applied, combined with the observation conditions and geometric constraints corresponding to each sample image, and the angle difference and position difference are used to reconstruct the position information of the reference marker in the three-dimensional coordinate system to form a three-dimensional spatial geometric relationship network, including: Applying an improved triangulation algorithm, combining observation conditions and geometric constraints corresponding to each sample image, and utilizing the angle difference and the position difference, calculates position information of the feature point pair in a three-dimensional coordinate system; By integrating the position information of all feature point pairs, the three-dimensional coordinates of multiple feature points on each reference marker are obtained; Determine the position information of each reference marker by fitting the three-dimensional coordinates of a plurality of feature points of each reference marker, wherein the position information includes the three-dimensional coordinates and the posture; Based on the position information of each reference marker, a three-dimensional spatial geometric relationship network is constructed to describe the relative angles and straight-line distances between all reference markers and between the reference markers and the target point. In the process of constructing the three-dimensional spatial geometric relationship network, the angle differences and position differences are used to correct possible errors.

4. The method according to claim 1, characterized in that: The method of using the three-dimensional spatial geometric relationship network to obtain the position information of the reference markers at the current level and the relative angles and straight-line distances between the reference markers, and calculating the preliminary position of the target point, includes: Using the three-dimensional spatial geometric relationship network, the position information of the reference markers and the relative angles between the reference markers under the current observation conditions are obtained to reproduce the observation conditions when the sample to be tested is first observed by the scanning electron microscope; Recovering the spatial geometric relationship between the target point and the reference markers based on the relative angle and straight-line distance of the target point relative to the reference marker recorded under the observation conditions when the sample to be tested is first observed under the scanning electron microscope, combined with the position information of the reference markers under the current observation conditions and the relative angle and straight-line distance between the reference markers; The spatial geometric relationship is used to calculate the preliminary position of the target point through an improved triangulation algorithm.

5. The method according to claim 4, characterized in that The step of obtaining the position information of the reference markers and the relative angles between the reference markers under the current observation conditions to reproduce the observation conditions when the sample to be tested is first observed by the scanning electron microscope includes: Based on the three-dimensional spatial geometric relationship network, obtaining the three-dimensional coordinates of each reference marker under the current observation condition as the position information of the reference marker; Calculating the relative angles between all reference markers in the same layer according to the position information to determine the relative angles between the reference markers under the current observation condition; The relative angle and straight-line distance between the reference markers and the angle of the sample stage under the current observation conditions are adjusted to ensure that the relative angle and straight-line distance between the reference markers are consistent with the relative angle and straight-line distance originally determined between the reference markers during the first observation, and the angle of the sample stage is consistent with the originally determined angle of the sample stage.

6. The method according to claim 1, characterized in that Also includes: When the sample to be tested is observed by a scanning electron microscope for the first time, multimodal data of the target point and the surrounding area are recorded and stored, wherein the multimodal data at least includes spectral characteristics and physical properties; When the sample to be tested is processed and subjected to a second SEM observation, multimodal data of the current target point and the surrounding area are recollected; By comparing the multimodal data stored during the first observation with the currently collected multimodal data, a multimodal feature matching result is generated, wherein the multimodal feature matching result includes change information of the target point and the surrounding area between the two observations; Based on the multimodal feature matching result, the target position of the target point is adjusted to ensure the positioning accuracy of the target point without being affected by the microstructure change of the sample to be tested.

7. The method according to claim 1, characterized in that The process of establishing a multi-level, multi-scale reference marker system around the target point, wherein each level of the reference marker system contains at least three non-collinear morphological features as reference markers, comprises: establishing a three-level reference marker system around the target point, and selecting, in the first level of the reference marker system, at least three non-collinear significant features of the overall structure of the sample to be tested on a macroscopic scale as reference markers, selecting, in the second level of the reference marker system, at least three non-collinear stable structures of the sample to be tested on a mesoscopic scale as reference markers, and selecting, in the third level of the reference marker system, at least three non-collinear fine features of the sample to be tested on a microscopic scale as reference markers.

8. The method according to claim 3, characterized in that The geometric constraints include a fundamental matrix; The method of applying the improved triangulation algorithm, combining the observation conditions and geometric constraints corresponding to each sample image, and using the angle difference and the position difference to calculate the position information of the feature point pair in the three-dimensional coordinate system includes: The position information of the feature point pair in the three-dimensional coordinate system is calculated by the following calculation formula, wherein the position information includes the three-dimensional coordinates; In' i =in i ·β i w i =exp(-α·||T i || 2 -β·||Δp i || 2 ) b i =exp(γ·θ i ) Among them, P is the three-dimensional coordinate of the feature point, n is the number of feature point pairs, i is the index of the feature point pair, and w' i is the comprehensive weight of the i-th group of feature point pairs, K i is the camera intrinsic parameter matrix corresponding to the i-th set of feature point pairs, x 1,i and x 2,i are the 2D coordinates of the same reference marker in two different sample images, and F i is the basic matrix corresponding to the i-th set of feature point pairs, w i is the initial weight of the i-th group of feature point pairs, β i is the exponential adjustment factor of the i-th group of feature point pairs, θ is the angle difference, Δp is the position difference, α is a positive adjustment parameter used to control the influence of the translation vector, T i is the translation vector corresponding to the i-th set of feature point pairs, ||T i || represents the Euclidean norm of the translation vector, β is a positive adjustment parameter used to control the influence of position difference, Δp i is the position difference of the i-th group of feature point pairs, indicating the relative displacement between two different observation perspectives, ||Δp i || 2 is the Euclidean norm of the position difference, γ is a positive adjustment parameter used to control the influence of the rotation angle, θ i is the angle difference of the i-th set of feature point pairs, indicating the relative rotation angle between two different observation perspectives.

9. The method according to claim 4, characterized in that The method of using the three-dimensional spatial geometric relationship network to obtain the position information of the reference markers at the current level and the relative angles and straight-line distances between the reference markers, and calculating the preliminary position of the target point, includes: Determine the position information of each reference marker in the three-dimensional coordinate system under the current observation conditions Calculate relative angles between reference markers and straight-line distance The linear distance d between the target point and the reference marker recorded under the observation conditions when the sample to be tested is first observed by the scanning electron microscope initial,ref , combined with the relative angles between the reference markers under the current observation conditions and straight-line distance Restore the spatial geometric relationship between the target point and the reference marker, and calculate the preliminary position P of the target point by the following formula initial : Where n is the number of reference markers; i is the index of the reference marker; is the weight of the i-th reference marker, according to the distance d from the target point to the reference marker under the first observation condition initial,ref The straight-line distance between the reference marker under the current observation conditions Determined, represents the straight-line distance between the i-th reference marker and all other reference markers j The sum of is the position information of the i-th reference marker in the three-dimensional coordinate system under the current observation conditions; ΔP i is the displacement vector of the target point relative to the i-th reference marker derived based on the spatial geometric relationship between the first observation condition and the current observation condition, and the displacement vector is calculated by the following formula: where θ initial,ij is the relative angle between the i-th reference marker and the j-th reference marker under the first observation condition; is the relative angle between the i-th reference marker and the j-th reference marker under the current observation condition; is the unit vector pointing from the i-th reference marker to the j-th reference marker.

10. A target point rapid positioning system based on scanning electron microscope observation, characterized in that: include: A construction module is used to select a target point on the sample to be tested when the sample to be tested is observed by a scanning electron microscope for the first time, and to establish a multi-level, multi-scale reference marker system around the target point, and each level of the reference marker system contains at least three non-collinear morphological features as reference markers, so as to ensure that the target point selected by the sample to be tested at different magnifications has a corresponding reference marker system as a positioning reference; an acquisition module, configured to adjust the angle of a sample stage on which the sample to be tested is placed, and each time the angle is adjusted and a sample image including the target point and reference markers around the target point is acquired through a scanning electron microscope, simultaneously record observation conditions when the sample image is acquired, including the angle of the sample stage and the magnification of the scanning electron microscope; A determination module, used for determining, from each sample image, the relative angle and straight-line distance between the target point and the reference markers selected at each level and between the reference markers; A generation module, configured to determine the angle difference and position difference of the same reference marker under different observation angles based on the multiple sample images, and reconstruct the position information of the reference marker in a three-dimensional coordinate system based on the angle difference and position difference through stereo vision technology to form a three-dimensional spatial geometric relationship network; A calculation module is used to determine the reference markers of the corresponding layer according to the current magnification when the sample to be tested is processed and subjected to a second scanning electron microscope observation, obtain the position information of the reference markers of the current layer and the relative angles between the reference markers by using the three-dimensional spatial geometric relationship network, and calculate the preliminary position of the target point; The determination module is also used to apply a local fine-tuning algorithm, combined with a sample image collected in real time, and move the sample stage at a sub-pixel level according to the preliminary position of the target point until the target point is located at the center of the field of view, so as to determine the target position of the target point, and during the movement of the sample stage, by monitoring and correcting possible system errors or sample displacements, ensure that the originally determined relative angles and straight-line distances between the reference markers remain consistent.

Images