Target point rapid positioning method and system based on scanning electron microscope observation

Abstract

The embodiment of the present application provides a target point fast positioning method and system based on scanning electron microscope observation. In the first observation, a target point is selected on a sample to be measured, and a multi-level reference marker system is established around the point. The relative angle and straight-line distance between the target point and the reference marker are determined by adjusting the sample stage angle and recording the observation conditions of each sample image. The position information of the reference marker in the three-dimensional coordinate system is reconstructed using the angle difference and position difference in multiple sample images, and a three-dimensional space geometric relationship network is formed. When the sample is treated and then observed by scanning electron microscope again, the preliminary position of the target point is calculated using the three-dimensional space geometric relationship network. The sample stage is moved at the sub-pixel level until the target point is located at the center of the field of view, and the system error or sample displacement is monitored and corrected. The technical scheme provided by the embodiment of the present application improves the accuracy, adaptability and efficiency of target point positioning.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and system for rapid target point localization based on scanning electron microscopy observation. Background Technology

[0002] In fields such as materials science, biology, and nanotechnology, scanning electron microscopy (SEM) is widely used as an important observation tool for the analysis of microstructures.

[0003] Currently, the common practice is to record the location coordinates of the target point during the initial scan, and to record as much information about the surrounding environment as possible to facilitate subsequent relocation. Another method is manual labeling, which involves adding easily identifiable artificial markers to the sample to aid in localization.

[0004] However, traditional positioning methods lack systematicity and stability. When the sample undergoes processing, the original coordinate information may become invalid. Manual marking may introduce additional variables and interfere with experimental results. Existing positioning methods usually do not take into account the changes in three-dimensional spatial position caused by the adjustment of the sample stage angle, which makes the repositioning of target points under complex observation conditions extremely difficult. Summary of the Invention

[0005] This application provides a method and system for rapid target point localization based on scanning electron microscopy observation, which solves the problems of low target point localization efficiency, lack of accuracy and adaptability in the prior art.

[0006] In a first aspect, embodiments of this application provide a method for rapid target point localization based on scanning electron microscopy observation, including:

[0007] When the sample to be tested is observed for the first time using a scanning electron microscope, a target point is selected on the sample to be tested, and a multi-level, multi-scale reference marker system is established around the target point. Each level of the reference marker system contains at least three non-collinear morphological features as reference markers to ensure that the target point selected on the sample to be tested at different magnifications has a corresponding reference marker system as a positioning benchmark.

[0008] By adjusting the angle of the sample stage on which the sample to be tested is placed, each time the angle is adjusted and a sample image containing the target point and the reference markers around the target point is acquired by scanning electron microscope, the observation conditions when acquiring the sample image are recorded, including the observation angle, the sample stage angle, and the magnification of the scanning electron microscope.

[0009] Determine the relative angles and straight-line distances between the target points and selected reference markers at each level, as well as between the reference markers, from each sample image;

[0010] Based on the multiple sample images, the angular and positional differences of the same reference marker under different observation perspectives are determined. Then, using stereo vision technology, the positional information of the reference marker in the three-dimensional coordinate system is reconstructed based on the angular and positional differences, forming a three-dimensional spatial geometric relationship network.

[0011] When the sample to be tested is processed and observed by a second scanning electron microscope, the reference markers at the corresponding level are determined according to the current magnification. The position information of the reference markers at the current level, the relative angles and straight-line distances between the reference markers are obtained using the three-dimensional spatial geometric relationship network, and the preliminary position of the target point is calculated.

[0012] By applying a local fine-tuning algorithm and combining it with real-time acquired sample images, the sample stage is moved at the sub-pixel level according to the initial position of the target point until the target point is located in the center of the field of view, thereby determining the target position of the target point. During the movement of the sample stage, any possible systematic errors or sample displacements are monitored and corrected to ensure that the originally determined relative angles and straight-line distances between the reference markers remain consistent.

[0013] Optionally, the step of determining the angular and positional differences of the same reference marker under different viewing angles based on the multiple sample images, and reconstructing the positional information of the reference marker in a three-dimensional coordinate system based on the angular and positional differences using stereo vision technology to form a three-dimensional spatial geometric relationship network, includes:

[0014] For each sample image, feature points of the reference marker are extracted using computer vision algorithms, and a descriptor is generated for each feature point. The feature points refer to key points in the sample image that can uniquely identify the position and shape of the reference marker.

[0015] By comparing feature descriptors in different sample images, feature points of the same reference marker from different observation perspectives are identified and feature point pairs are generated. Each feature point pair consists of one or more pairs of feature points from different sample images. The feature points in each feature point pair have similar feature descriptors and correspond to the same physical location.

[0016] Based on the feature point pairs, geometric constraints are applied to estimate the relative geometric relationship between different observation viewpoints, and the rotation and translation changes of the camera on the extrinsic parameters are calculated by decomposing the relative geometric relationship to obtain the camera attitude information.

[0017] Based on the feature point pairs and the camera pose information, calculate the relative rotation angle of the same reference marker under different viewing angles and use it as the angle difference; calculate the relative displacement of the same reference marker under different viewing angles and use it as the position difference.

[0018] By applying an improved triangulation algorithm, combined with the observation conditions and geometric constraints corresponding to each sample image, and utilizing the angular and positional differences, the positional information of the reference marker in the three-dimensional coordinate system is reconstructed, forming a three-dimensional spatial geometric relationship network.

[0019] Optionally, the improved triangulation algorithm, combined with the observation conditions and geometric constraints corresponding to each sample image, utilizes the angle and position differences to reconstruct the position information of the reference marker in the three-dimensional coordinate system, forming a three-dimensional spatial geometric relationship network, including:

[0020] An improved triangulation algorithm is applied, and combined with the observation conditions and geometric constraints corresponding to each sample image, the position information of the feature point pair in the three-dimensional coordinate system is calculated using the angle difference and the position difference.

[0021] By integrating the positional information of all feature point pairs, the three-dimensional coordinates of multiple feature points on each reference marker are obtained;

[0022] The position information of each reference marker is determined by fitting the three-dimensional coordinates of multiple feature points of each reference marker. The position information includes three-dimensional coordinates and orientation.

[0023] Based on the position information of each reference marker, a three-dimensional spatial geometric relationship network is constructed to describe the relative angles and straight-line distances between all reference markers and between reference markers and target points. In the process of constructing the three-dimensional spatial geometric relationship network, the angle differences and position differences are used to correct any possible errors.

[0024] Optionally, the step of using the three-dimensional spatial geometric relationship network to obtain the position information of the reference markers at the current level, the relative angles and straight-line distances between the reference markers, and to calculate the preliminary position of the target point includes:

[0025] Using the three-dimensional spatial geometric relationship network, the position information of the reference markers and the relative angles between the reference markers under the current observation conditions are obtained, so as to reproduce the observation conditions when the sample under test was first observed by scanning electron microscopy;

[0026] Based on the relative angle and straight-line distance of the target point relative to the reference marker recorded under the observation conditions when the sample under test was first observed by scanning electron microscopy, and combined with the position information of the reference marker under the current observation conditions and the relative angle and straight-line distance between the reference markers, the spatial geometric relationship between the target point and the reference marker is restored.

[0027] Using the aforementioned spatial geometric relationships and an improved triangulation algorithm, the preliminary position of the target point is calculated.

[0028] Optionally, obtaining the position information of the reference markers and the relative angles between the reference markers under the current observation conditions to reproduce the observation conditions when the sample to be tested was first observed by scanning electron microscopy includes:

[0029] Based on the three-dimensional spatial geometric relationship network, the three-dimensional coordinates of each reference marker under the current observation conditions are obtained as the position information of the reference marker;

[0030] Based on the location information, calculate the relative angles between all reference markers at the same level to determine the relative angles between reference markers under the current observation conditions;

[0031] The relative angles and straight-line distances between reference markers and the angle of the sample stage under the current observation conditions are adjusted to ensure that the relative angles and straight-line distances between reference markers are consistent with the original relative angles and straight-line distances between reference markers during the first observation, and that the angle of the sample stage is consistent with the original determined angle of the sample stage.

[0032] Optionally, it also includes:

[0033] When the sample to be tested is observed for the first time using a scanning electron microscope, multimodal data of the target point and its surrounding area are recorded and stored. The multimodal data includes at least spectral features and physical properties.

[0034] When the sample to be tested is processed and subjected to a second scanning electron microscope observation, multimodal data of the current target point and surrounding area are re-acquired;

[0035] By comparing the multimodal data stored during the first observation with the currently acquired multimodal data, a multimodal feature matching result is generated. The multimodal feature matching result includes information on the changes in the target point and its surrounding area between the two observations.

[0036] Based on the multimodal feature matching results, the target position of the target point is adjusted to ensure the accuracy of the target point's positioning and to ensure that it is not affected by changes in the microstructure of the sample under test.

[0037] Optionally, the process of establishing a multi-level, multi-scale reference marker system around the target point, wherein each level of the reference marker system contains at least three non-collinear morphological features as reference markers, includes: establishing a three-layer reference marker system around the target point, and selecting significant features of the overall structure of the sample under test that contain at least three non-collinear features at the macroscopic scale as reference markers in the first layer of the reference marker system; selecting stable structures of the sample under test that contain at least three non-collinear features at the mesoscopic scale as reference markers in the second layer of the reference marker system; and selecting fine features of the sample under test that contain at least three non-collinear features at the microscopic scale as reference markers in the third layer of the reference marker system.

[0038] Optionally, the geometric constraints include a fundamental matrix;

[0039] The improved triangulation algorithm, combined with the observation conditions and geometric constraints corresponding to each sample image, calculates the position information of the feature point pair in the three-dimensional coordinate system using the angle difference and the position difference, including:

[0040] The position information of feature point pairs in the three-dimensional coordinate system is calculated using the following formula, where the position information includes three-dimensional coordinates;

[0041]

[0042] w′ i =w i ·β i

[0043] w i =exp(-α·||T) i || 2 -β·||Δp i || 2 )

[0044] β i =exp(γ·θ) i )

[0045] Where P is the 3D coordinate of the feature point, n is the number of feature point pairs, i is the index of the feature point pair, and w' i K is the combined weight of the i-th feature point pair. i x is the camera intrinsic parameter matrix corresponding to the i-th pair of feature points. 1,i and x 2,i These are the two-dimensional coordinates of the same reference marker in two different sample images, F i Let be the fundamental matrix corresponding to the i-th feature point pair, wi be the initial weight of the i-th feature point pair, and β be the initial weight of the i-th feature point pair. iα is the exponential adjustment factor for the i-th feature point pair, θ is the angular difference, Δp is the positional difference, and α is a positive adjustment parameter used to control the degree of influence of the translation vector. i It is the translation vector corresponding to the i-th pair of feature points, ||T i || represents the Euclidean norm of the translation vector, β is a positive adjustment parameter used to control the degree of influence of positional differences, and Δp i The positional difference of the i-th pair of feature points represents the relative displacement between two different observation viewpoints, ||Δp i || 2 γ is the Euclidean norm of the positional difference, γ is a positive adjustment parameter used to control the degree of influence of the rotation angle, and θ is the positional difference. i It is the angular difference of the i-th pair of feature points, representing the relative rotation angle between two different observation viewpoints.

[0046] Optionally, the step of using the three-dimensional spatial geometric relationship network to obtain the position information of the reference markers at the current level, the relative angles and straight-line distances between the reference markers, and to calculate the preliminary position of the target point includes:

[0047] Determine the position information of each reference marker in the three-dimensional coordinate system under the current observation conditions.

[0048] Calculate the relative angles between reference markers and straight-line distance

[0049] The straight-line distance d of the target point relative to the reference marker, recorded under the observation conditions during the first scanning electron microscope observation of the sample under test. initial,ref Combined with the relative angles between reference markers under the current observation conditions and straight-line distance Restore the spatial geometric relationship between the target point and the reference marker, and calculate the preliminary position p of the target point using the following formula. initial :

[0050]

[0051] Where n is the number of reference markers; i is the index of the reference marker; It is the weight of the i-th reference marker, based on the distance d from the target point to the reference marker under the initial observation conditions. initial,ref The straight-line distance between the reference marker and the current observation conditions The determination, This represents the straight-line distance between the i-th reference marker and all other reference markers j. The sum; It represents the position information of the i-th reference marker in the three-dimensional coordinate system under the current observation conditions; ΔP i The displacement vector of the target point relative to the i-th reference marker is derived based on the spatial geometric relationship between the initial observation conditions and the current observation conditions. The displacement vector is calculated using the following formula:

[0052]

[0053] Where θ initial,ij It is the relative angle between the i-th and j-th reference markers under the initial observation conditions; It is the relative angle between the i-th and j-th reference markers under the current observation conditions; It is the unit vector pointing from the i-th reference marker to the j-th reference marker.

[0054] Secondly, embodiments of this application provide a rapid target point localization system based on scanning electron microscopy observation, comprising:

[0055] The module is used to select a target point on the sample to be tested when the sample is first observed by scanning electron microscopy, and to establish a multi-level, multi-scale reference marker system around the target point. Each level of the reference marker system contains at least three non-collinear morphological features as reference markers to ensure that the target point selected by the sample to be tested at different magnifications has a corresponding reference marker system as a positioning reference.

[0056] The acquisition module is used to adjust the angle of the sample stage on which the sample to be tested is placed, and to record the observation conditions when acquiring the sample image by a scanning electron microscope each time the angle is adjusted and the sample image containing the target point and the reference markers around the target point is acquired, including the angle of the sample stage and the magnification of the scanning electron microscope.

[0057] A determination module is used to determine, from each sample image, the relative angles and straight-line distances between the target points and selected reference markers at each level, as well as between the reference markers;

[0058] The generation module is used to determine the angular and positional differences of the same reference marker under different observation angles based on the multiple sample images, and to reconstruct the positional information of the reference marker in the three-dimensional coordinate system based on the angular and positional differences using stereo vision technology, thereby forming a three-dimensional spatial geometric relationship network.

[0059] The calculation module is used to determine the reference markers at the corresponding level according to the current magnification when the sample to be tested is processed and observed by a second scanning electron microscope. It uses the three-dimensional spatial geometric relationship network to obtain the position information of the reference markers at the current level and the relative angles between the reference markers, and calculates the preliminary position of the target point.

[0060] The determining module is also used to apply a local fine-tuning algorithm, combined with the real-time acquired sample images, to move the sample stage at the sub-pixel level according to the preliminary position of the target point, until the target point is located in the center of the field of view, so as to determine the target position of the target point. During the movement of the sample stage, by monitoring and correcting possible system errors or sample displacement, it ensures that the originally determined relative angle and straight-line distance between the reference markers remain consistent.

[0061] Thirdly, embodiments of this application provide a computing device, including a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are to be invoked and executed by the processing component to realize a method for rapid target point localization based on scanning electron microscopy observation as described in the first aspect above.

[0062] Fourthly, embodiments of this application provide a computer storage medium storing a computer program, which, when executed by a computer, implements a method for rapid target point localization based on scanning electron microscopy observation as described in the first aspect.

[0063] In this embodiment, when the sample to be tested is observed for the first time using a scanning electron microscope (SEM), a target point is selected on the sample, and a multi-level, multi-scale reference marker system is established around the target point. Each level of the reference marker system contains at least three non-collinear morphological features as reference markers to ensure that the target point selected at different magnifications has a corresponding reference marker system as a positioning benchmark. By adjusting the angle of the sample stage on which the sample is placed, each time the angle is adjusted and a sample image containing the target point and the reference markers around the target point is acquired using the SEM, the observation conditions when acquiring the sample image are recorded, including the observation angle, the sample stage angle, and the SEM magnification. The relative angles and straight-line distances between the target point and the selected reference markers at each level, as well as between the reference markers, are determined from each sample image. Based on the multiple sample images, the same reference marker is determined at different magnifications. The system measures angular and positional differences from different viewing angles and uses stereo vision technology to reconstruct the positional information of the reference markers in a three-dimensional coordinate system based on these differences, forming a three-dimensional spatial geometric relationship network. When the sample under test is processed and observed with a second scanning electron microscope, the reference markers at the corresponding level are determined according to the current magnification. Using the three-dimensional spatial geometric relationship network, the positional information of the reference markers at the current level, as well as the relative angles and straight-line distances between the reference markers, are obtained, and the preliminary position of the target point is calculated. Applying a local fine-tuning algorithm, combined with real-time acquired sample images, the sample stage is moved at the sub-pixel level according to the preliminary position of the target point until the target point is located at the center of the field of view, thus determining the target position of the target point. During the movement of the sample stage, any possible systematic errors or sample displacements are monitored and corrected to ensure that the originally determined relative angles and straight-line distances between the reference markers remain consistent.

[0064] The technical solution of this application has the following beneficial effects:

[0065] This application selects a target point during the initial scanning electron microscope (SEM) observation and establishes a multi-layered, multi-scale reference marker system around it, ensuring a reliable positioning benchmark at different magnifications. This method meticulously records the observation conditions after each adjustment of the sample stage angle and uses stereo vision technology to reconstruct the positional information of the reference markers in the three-dimensional coordinate system, forming a precise spatial geometric relationship network. When the sample needs to be observed again after processing, the corresponding level of reference markers can be quickly determined based on the current magnification, and the preliminary position of the target point is calculated using the pre-constructed three-dimensional spatial geometric relationship network. Subsequently, a local fine-tuning algorithm is applied for sub-pixel-level precise positioning until the target point is accurately located at the center of the field of view. Throughout the process, potential systematic errors or sample displacements are monitored and corrected in real time to ensure the consistency and accuracy of the relative positions between reference markers. This method significantly improves the speed and accuracy of target point repositioning in SEM, solving the problem of repeated positioning caused by sample changes that traditional methods struggle with, especially maintaining high efficiency and accuracy under high-resolution conditions. It not only enhances the stability and reliability of positioning but also supports the analysis of complex structures, providing a powerful tool for research in materials science, biology, and nanotechnology. By enabling rapid and accurate target point localization, this method significantly improves the efficiency of microstructure analysis, simplifies the experimental process, and lays a solid foundation for subsequent research and applications.

[0066] Furthermore, this embodiment of the application utilizes computer vision algorithms to extract feature points and generate descriptors for reference markers in multiple sample images. By comparing feature descriptors from different viewing angles, feature point pairs with the same physical location are identified. Based on these feature point pairs, geometric constraints are applied to estimate the relative geometric relationships between different viewing angles, and camera pose information is calculated. Further, an improved triangulation algorithm, combined with observation conditions and geometric constraints, reconstructs the position information of the reference markers in the three-dimensional coordinate system using angle and position differences, forming a precise three-dimensional spatial geometric relationship network. In this process, the position information of all feature points is integrated to determine the three-dimensional coordinates and pose of each reference marker, and a three-dimensional spatial geometric relationship network describing the relative relationships between all reference markers and between reference markers and target points is constructed, while correcting for any potential errors.

[0067] The above method significantly enhances the accuracy and reliability of target point localization in scanning electron microscopy. By accurately extracting and matching feature points, and precisely calculating the geometric relationships between different observation perspectives, a high-precision three-dimensional spatial geometric relationship network can be reconstructed even under complex conditions. The application of improved triangulation algorithms enables more accurate determination of the three-dimensional position and orientation of reference markers, thereby improving the accuracy of the entire localization system. Furthermore, real-time error correction during the construction of the three-dimensional spatial geometric relationship network further ensures the stability and consistency of the localization results, providing a solid foundation for subsequent precise target point localization. This method not only improves work efficiency but also makes microstructure analysis simpler and more efficient, bringing significant value to scientific research and technological applications.

[0068] These or other aspects of this application will become more apparent in the following description of the embodiments. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0070] Figure 1 A flowchart of a method for rapid target point localization based on scanning electron microscopy observation provided in this application is shown;

[0071] Figure 2 This paper presents a schematic diagram of a target point rapid localization system based on scanning electron microscopy observation provided in this application.

[0072] Figure 3 A schematic diagram of the structure of a computing device provided in this application is shown. Detailed Implementation

[0073] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0074] In some of the processes described in the specification, claims, and accompanying drawings of this application, multiple operations appearing in a specific order are included. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not themselves represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a chronological order, nor do they limit "first" and "second" to different types.

[0075] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0076] Figure 1 This application provides a flowchart of a method for rapid target point localization based on scanning electron microscopy observation, as shown in the embodiments. Figure 1 As shown, the method includes:

[0077] 101. When the sample to be tested is observed for the first time by scanning electron microscopy, a target point is selected on the sample to be tested, and a multi-level, multi-scale reference marker system is established around the target point. Each level of the reference marker system contains at least three non-collinear morphological features as reference markers to ensure that the target point selected by the sample to be tested at different magnifications has a corresponding reference marker system as a positioning reference.

[0078] The target point refers to a specific location on the sample to be tested that has clear characteristics and research value, and it is the core object of subsequent observation and analysis.

[0079] In a multi-level, multi-scale reference marker system, "multi-level" refers to the positioning references established at different magnifications; "multi-scale" means that these references include various morphological features from macroscopic to microscopic. The three non-collinear morphological features in each level ensure that the target point can be stably located even at different magnifications using these invariant reference points.

[0080] The process of establishing a multi-level, multi-scale reference marker system around the target point, wherein each level of the reference marker system contains at least three non-collinear morphological features as reference markers, includes: establishing a three-layer reference marker system around the target point; selecting significant features of the overall structure of the sample under test (such as edge corners, cracks, etc.) containing at least three non-collinear features at the macroscopic scale as reference markers in the first layer of the reference marker system; selecting stable structures of the sample under test (such as pores, mineral grains) containing at least three non-collinear features at the mesoscopic scale as reference markers in the second layer of the reference marker system; and selecting fine features of the sample under test (such as crystal defects) containing at least three non-collinear features at the microscopic scale as reference markers in the third layer of the reference marker system.

[0081] In practice, a target point with distinct characteristics is selected, and a series of markers with significant morphological features are chosen around it. These markers need to have sufficient contrast and stability to be identifiable at different magnifications. In this way, even if the sample is processed or its surface features change, the target point can be re-found based on these invariant reference markers.

[0082] For example, in the field of artificial intelligence, machine learning algorithms can be used to automatically identify and label these reference markers. Training models can learn representations of different morphological features from a large amount of labeled SEM image data, thereby automatically selecting appropriate reference markers on new samples. This not only improves efficiency but also reduces human error.

[0083] 102. By adjusting the angle of the sample stage on which the sample to be tested is placed, and each time the angle is adjusted and a sample image containing the target point and the reference markers around the target point is acquired using a scanning electron microscope, the observation conditions when acquiring the sample image are simultaneously recorded.

[0084] Whenever the sample stage angle changes, the system automatically acquires a new sample image and simultaneously records the observation conditions at that time. These conditions include, at a minimum, the observation angle, the sample stage angle, and the magnification of the scanning electron microscope. This information is stored along with the image so that the observation environment can be accurately reproduced in subsequent analyses.

[0085] The observation angle refers to the spatial position and orientation of the scanning electron microscope (SEM) relative to the sample, determining the directionality of the imaging and the content of the acquired image. Different observation angles can reveal different sides or internal structures of the sample. The stage angle refers to the orientation of the stage relative to a fixed reference frame (such as the laboratory coordinate system), used to change the observation direction of the sample, thereby acquiring image information from more angles. Magnification is one of the operating parameters of an SEM, determining the size and level of detail of the sample area displayed in the image. High magnification can reveal finer structures, while low magnification covers a larger area.

[0086] Continuing with the example above, the AI ​​system can integrate an automated control module to automatically rotate the sample stage and capture images according to a preset angle sequence. Simultaneously, sensor technology can be used to monitor and record observation conditions in real time, such as environmental factors like temperature and humidity, ensuring consistency of experimental conditions. Furthermore, deep learning models can be introduced to predict the optimal observation angle, optimizing the image acquisition process.

[0087] 103. Determine the relative angles and straight-line distances between the target points and the selected reference markers at each level, as well as between the reference markers, from each sample image;

[0088] Relative angle refers to the angular relationship between the target point and the reference markers, as well as between the reference markers themselves, and is an important parameter for constructing geometric models.

[0089] Straight-line distance is the actual physical distance between the target point and the reference markers, as well as between the reference markers themselves, used to accurately describe their spatial relationships.

[0090] By analyzing each sample image, the relative positional relationship between the target point and a selected reference marker is calculated, including angle and distance. This process may involve image processing techniques and computer vision algorithms.

[0091] At this stage of the above embodiments, advanced computer vision algorithms, such as SIFT (Scale Invariant Feature Transform) or ORB (Oriented Fast and Rotated BREF), can be applied to detect and match feature points, thereby calculating accurate geometric relationships. Combining deep learning methods can make this process more efficient and accurate.

[0092] 104. Based on the multiple sample images, determine the angular and positional differences of the same reference marker under different viewing angles, and reconstruct the positional information of the reference marker in the three-dimensional coordinate system based on the angular and positional differences using stereo vision technology, thereby forming a three-dimensional spatial geometric relationship network.

[0093] Angular and positional differences refer to the changes in rotation angle and spatial displacement of the same reference marker under different observation angles, reflecting the geometric transformation between different observation angles.

[0094] Stereo vision technology is a technique that uses images from two or more different viewing angles to reconstruct a three-dimensional scene. By comparing the differences between images from different viewing angles, the true three-dimensional coordinates of the object can be calculated.

[0095] The three-dimensional spatial geometric relationship network, consisting of a three-dimensional coordinate system composed of all reference markers and their relationships with each other, provides a complete description of the sample structure and supports subsequent precise positioning.

[0096] By utilizing feature point pairs in multi-view images, the true three-dimensional coordinates of each reference marker are estimated through geometric methods such as triangulation, thus constructing a spatial geometric network that describes the relationships between all reference markers.

[0097] For example, in the field of AI, technologies such as structured light scanning, binocular vision, or multi-view geometry can be used to achieve high-precision 3D reconstruction. Furthermore, traditional triangulation algorithms can be improved by training neural network models, thereby increasing the accuracy and speed of reconstruction. For instance, convolutional neural networks (CNNs) can be used to extract more robust feature points, thus better handling noise and complex backgrounds.

[0098] 105. When the sample to be tested is processed and observed by a second scanning electron microscope, the reference markers at the corresponding level are determined according to the current magnification. The position information of the reference markers at the current level and the relative angles and straight-line distances between the reference markers are obtained using the three-dimensional spatial geometric relationship network, and the preliminary position of the target point is calculated.

[0099] The preliminary location, estimated based on current observation conditions and existing three-dimensional geometric relationships, provides a starting point for the next step of precise positioning.

[0100] Based on a pre-established three-dimensional spatial geometric network and current observation conditions, the region where the target point is located is quickly identified, and its approximate location is calculated. This step simplifies the subsequent fine-tuning process and improves positioning efficiency.

[0101] Building upon the above embodiments, the AI ​​system can intelligently select the closest reference marker layer based on current observation conditions and quickly estimate the target point's location by comparing real-time images with models in the database. By continuously optimizing the selection strategy using machine learning algorithms, the localization process becomes both fast and accurate.

[0102] 106. By applying a local fine-tuning algorithm and combining it with real-time acquired sample images, the sample stage is moved at the sub-pixel level according to the initial position of the target point until the target point is located in the center of the field of view, so as to determine the target position of the target point. During the movement of the sample stage, possible systematic errors or sample displacements are monitored and corrected to ensure that the originally determined relative angles and straight-line distances between the reference markers remain consistent.

[0103] The local fine-tuning algorithm is an optimization algorithm used to make fine adjustments to the sample stage to achieve sub-pixel level precise control and ensure that the target point is ultimately located in the center of the field of view.

[0104] Subpixel level refers to a level of precision smaller than a pixel, and is typically used in image processing and machine vision to achieve higher positioning accuracy.

[0105] Systematic errors or sample displacement refer to measurement deviations or sample movement caused by factors such as wear of mechanical parts and temperature changes. These errors need to be monitored and corrected in real time to ensure positioning accuracy.

[0106] Based on the initial positioning, the sample stage position is further adjusted to ensure the target point is precisely centered in the field of view. This process requires continuous comparison of real-time images with the expected position, gradually reducing errors until the requirements are met.

[0107] Building upon the above embodiments, by introducing reinforcement learning algorithms, the AI ​​system can autonomously learn how to optimally perform sub-pixel-level adjustments. The system continuously optimizes its adjustment strategy based on feedback from each adjustment, achieving the best positioning results. Simultaneously, a closed-loop control system enables real-time error correction, ensuring the stability and reliability of the entire positioning process. Thus, from the initial selection of reference markers to the final precise positioning of the target point, the entire process can be completed by an intelligent AI system, significantly improving work efficiency and accuracy.

[0108] By implementing steps 101 to 106, the speed, accuracy, and reliability of target point localization in scanning electron microscopy (SEM) are significantly improved. First, a multi-level, multi-scale reference marker system is established during the initial observation (step 101), ensuring reliable positioning benchmarks for the target point at different magnifications, providing a solid foundation for subsequent observations. Next, by adjusting the sample stage angle and recording detailed observation conditions (step 102), not only is rich sample image data acquired, but consistency and repeatability of each observation are also guaranteed. Based on this, the relative geometric relationship between the target point and the reference markers is determined (step 103), laying the foundation for constructing an accurate spatial model. Subsequently, the three-dimensional coordinates of the reference markers are reconstructed using stereo vision technology and an improved triangulation algorithm (step 104), forming a complete spatial geometric relationship network. This step greatly improves the stability and accuracy of the localization system, enabling high-precision three-dimensional reconstruction even under complex conditions. When the sample needs to be observed again after processing, the corresponding reference marker layer is selected according to the current magnification, and the preliminary position of the target point is calculated using the existing three-dimensional geometric relationship network (step 105). This process quickly narrows the search range and greatly simplifies the final precise positioning process. Finally, a local fine-tuning algorithm is applied to move the sample stage at the sub-pixel level until the target point is located at the center of the field of view (step 106). Throughout the process, any possible systematic errors or sample displacements are monitored and corrected in real time, ensuring a high degree of consistency and accuracy in the positioning results. This method not only enhances the stability of positioning but also supports the analysis of complex structures, providing a powerful tool for research in materials science, biology, and nanotechnology. By achieving rapid and accurate target point positioning, this method significantly improves the efficiency of microstructure analysis, simplifies the experimental process, and lays a solid foundation for subsequent research and applications. In summary, from the initial establishment of the reference marker system to the final sub-pixel level fine-tuning, each step is interconnected, forming an efficient and accurate scanning electron microscope target point positioning solution, bringing significant value to scientific research and technological applications.

[0109] To address the difficulty in accurately reconstructing the 3D position information of reference markers in traditional methods and further improve the accuracy and reliability of target point localization, in some embodiments, step 104 involves determining the angular and positional differences of the same reference marker under different viewing angles based on the multiple sample images, and then reconstructing the position information of the reference marker in the 3D coordinate system based on the angular and positional differences using stereo vision technology, forming a 3D spatial geometric relationship network, including:

[0110] For each sample image, feature points of the reference marker are extracted using computer vision algorithms, and a descriptor is generated for each feature point. The feature points are key points in the sample image that uniquely identify the position and shape of the reference marker. By comparing the feature descriptors in different sample images, feature points of the same reference marker from different viewing angles are identified, and feature point pairs are generated. Each feature point pair consists of one or more pairs of feature points from different sample images. The feature points in each pair have similar feature descriptors and correspond to the same physical location. Based on the feature point pairs, geometric constraints are applied to estimate the relative geometric relationship between different viewing angles. The rotation and translation changes of the camera in the extrinsic parameters are calculated by decomposing the relative geometric relationship to obtain camera attitude information. Based on the feature point pairs and the camera attitude information, the relative rotation angle of the same reference marker under different viewing angles is calculated as the angle difference, and the relative displacement of the same reference marker under different viewing angles is calculated as the position difference. An improved triangulation algorithm is applied, combined with the observation conditions and geometric constraints corresponding to each sample image, and using the angle and position differences, the position information of the reference marker in the three-dimensional coordinate system is reconstructed, forming a three-dimensional spatial geometric relationship network.

[0111] In this embodiment, feature points are key points in the sample image that can uniquely identify the location and shape of the reference marker. These points typically have high contrast or unique geometric properties, enabling them to be reliably identified in images from different viewing angles.

[0112] A descriptor is a numerical representation generated for each feature point, used to describe the local image information surrounding that point. Descriptors enable feature points at the same physical location to be matched and identified even from different viewing perspectives.

[0113] A feature point pair consists of one or more pairs of feature points from different sample images. The feature points in each pair have similar feature descriptors and correspond to the same physical location. Feature point pairs are the foundation for constructing three-dimensional geometric relationships.

[0114] Camera pose information includes the camera's rotation and translation relative to the world coordinate system, i.e., extrinsic parameters. It describes the relative geometric relationship between different viewing angles and is crucial for reconstructing 3D coordinates.

[0115] The specific process of "calculating the relative rotation angle of the same reference marker under different observation views based on the feature point pair and the camera attitude information and using it as the angle difference, and calculating the relative displacement of the same reference marker under different observation views as the position difference" can be achieved by comparing the included angle and the difference in straight-line distance between feature point pairs under two observation views.

[0116] An improved triangulation algorithm is an optimized algorithm that combines observation conditions and geometric constraints corresponding to multiple sample images to accurately reconstruct the position of a reference marker in a three-dimensional coordinate system by calculating the angular and positional differences of feature point pairs.

[0117] In this embodiment, a computer vision algorithm is first used to extract feature points of reference markers from each sample image, and a descriptor is generated for each feature point. Then, by comparing the feature descriptors in different sample images, feature points belonging to the same physical location are identified and feature point pairs are formed. Based on these feature point pairs, geometric constraints are applied to estimate the relative geometric relationships between different viewing angles, and the camera's pose information (rotation and translation) is calculated.

[0118] Subsequently, based on feature point pairs and camera pose information, the relative rotation angle and displacement of the same reference marker under different observation angles are calculated as angular differences and positional differences.

[0119] Finally, by utilizing an improved triangulation algorithm combined with observation conditions and geometric constraints, and through the aforementioned angular and positional differences, the positional information of the reference marker in the three-dimensional coordinate system is accurately reconstructed, forming a complete three-dimensional spatial geometric relationship network. Throughout this process, potential errors are continuously corrected to ensure a high degree of consistency in the positioning results.

[0120] Here is a specific example:

[0121] In a specific embodiment within the field of artificial intelligence, deep learning techniques can be employed to enhance the accuracy and efficiency of the above steps. Suppose an automated scanning electron microscope (SEM) target point localization system is being developed. First, a convolutional neural network (CNN) model, such as U-Net or ResNet, is trained specifically for feature point detection and descriptor generation in SEM images. This model is trained using a large amount of labeled SEM image data to learn the optimal representation of different morphological features, thereby improving the accuracy of feature point matching. Next, an attention-based matching network, such as Transformer, is developed to automatically identify and match feature point pairs from different viewing angles. This network can handle complex background interference, further improving matching accuracy.

[0122] To estimate the relative geometric relationships between different viewing angles, a graph neural network (GNN) is used to model the geometric relationships between feature point pairs and predict camera pose information. GNNs can capture the complex interactions between feature point pairs, providing more accurate geometric relationship estimates. Then, an end-to-end deep learning framework, such as Pix2Vox or PointNet++, is implemented to directly reconstruct the 3D model of the reference marker from the multi-view images. This approach not only improves reconstruction speed but also generates more refined and accurate 3D structures.

[0123] Throughout the process, reinforcement learning algorithms, such as DQN or PPO, are introduced to dynamically adjust errors that occur during reconstruction. The system continuously optimizes the error correction strategy based on feedback from each adjustment, ensuring that the final 3D spatial geometric relationship network is as close to reality as possible. This approach not only achieves a highly automated process but also utilizes advanced AI technology at each stage, significantly improving the accuracy and efficiency of target point localization in scanning electron microscopy, providing strong support for scientific research and technological applications.

[0124] To address the difficulty in accurately reconstructing the 3D coordinates and orientation of reference markers using traditional methods, and to further improve the accuracy and stability of target point localization, some embodiments employ an improved triangulation algorithm. This algorithm, combined with the observation conditions and geometric constraints corresponding to each sample image, utilizes the angular and positional differences to reconstruct the position information of the reference marker in the 3D coordinate system, forming a 3D spatial geometric relationship network. This includes:

[0125] An improved triangulation algorithm is applied, combining the observation conditions and geometric constraints corresponding to each sample image, to calculate the position information of the feature point pairs in the three-dimensional coordinate system using the angle differences and position differences. By integrating the position information of all feature point pairs, the three-dimensional coordinates of multiple feature points on each reference marker are obtained. By fitting the three-dimensional coordinates of multiple feature points on each reference marker, the position information of each reference marker is determined, including the three-dimensional coordinates and orientation. Based on the position information of each reference marker, a three-dimensional spatial geometric relationship network describing the relative angles and straight-line distances between all reference markers and between the reference markers and the target point is constructed. During the construction of the three-dimensional spatial geometric relationship network, the angle differences and position differences are used to correct any possible errors.

[0126] In this scheme, the position information of each reference marker is determined by fitting the three-dimensional coordinates of multiple feature points of each reference marker because multiple feature points together describe the spatial distribution of the reference marker, and therefore the three-dimensional coordinates of multiple feature points can represent the position information of the reference marker in the three-dimensional coordinate system.

[0127] In this embodiment, the improved triangulation algorithm is an optimized geometric reconstruction method that combines observation conditions (such as magnification and viewing angle) and geometric constraints (such as known camera parameters) corresponding to multiple sample images to accurately reconstruct the position of the reference marker in the three-dimensional coordinate system by calculating the angular and positional differences of feature point pairs. Compared with traditional triangulation, the improved algorithm can generally handle noise better, reduce error accumulation, and improve reconstruction accuracy.

[0128] Angular and positional differences refer to the changes in rotation angle and spatial displacement of the same reference marker under different observation perspectives. These differences reflect the geometric transformations between different observation perspectives and are one of the key pieces of information for constructing a 3D model.

[0129] The location information of a feature point pair refers to the specific coordinates of the feature point pairs extracted and successfully matched from multiple sample images in a three-dimensional coordinate system. The feature points in each feature point pair have similar descriptors and correspond to the same physical location. Their location information is used to determine the overall three-dimensional structure of the reference marker.

[0130] Three-dimensional coordinates and pose: Three-dimensional coordinates represent the precise position of a reference marker in space; pose describes its orientation relative to the world coordinate system, including rotation and translation. This information is crucial for understanding the spatial distribution of reference markers and their relative relationships with other objects.

[0131] In this embodiment, to more accurately reconstruct the three-dimensional coordinates and orientation of the reference marker, this scheme employs an improved triangulation algorithm. This algorithm combines the observation conditions and geometric constraints of each sample image, utilizing angle and position differences for calculation. First, the improved triangulation algorithm is applied to calculate the accurate position of each pair of feature points in the three-dimensional coordinate system based on the angle and position differences.

[0132] Next, the positional information of all feature point pairs is aggregated to obtain the 3D coordinates of multiple feature points on each reference marker, ensuring that the reconstructed 3D model has sufficient detail and accuracy. Then, by fitting the 3D coordinates of multiple feature points of each reference marker, the overall positional information of the reference marker, including its 3D coordinates and orientation, is determined. This step helps to eliminate potential errors from individual feature points and provides a more stable positioning reference.

[0133] Finally, based on the positional information of each reference marker, a complete three-dimensional spatial geometric relationship network is constructed, describing the relative angles and straight-line distances between all reference markers and between reference markers and the target point. During this process, potential errors are corrected using angle and positional differences to ensure the high consistency and reliability of the entire network.

[0134] Here is a specific example:

[0135] Suppose we are developing an automated scanning electron microscope (SEM) target point localization system. To enhance the accuracy and efficiency of the above steps, we can train a convolutional neural network (CNN), such as ResNet or EfficientNet, specifically for feature point detection and descriptor generation in SEM images. This model can be trained on a large amount of labeled SEM image data to learn the optimal representation of different morphological features, thereby improving the accuracy of feature point matching. For example, we can use a pre-trained ResNet model as a base and add SEM image-specific fine-tuning layers to adapt to different material and structural properties.

[0136] Develop an attention-based matching network, such as the Transformer, to automatically identify and match feature point pairs from different viewing perspectives. This network can handle complex background noise, further improving matching accuracy. For example, the self-attention mechanism in the Transformer model can be used to enhance the correlation between feature point pairs, thereby more reliably identifying the same feature points from different viewing perspectives.

[0137] Graph Neural Networks (GNNs) are used to model the geometric relationships between feature point pairs to predict camera pose information. GNNs can capture the complex interactions between feature point pairs, providing more accurate geometric relationship estimates. For example, using GNN models such as GraphSAGE or GAT (Graph Attack Networks), a graph structure is built based on feature point pairs, node features are updated through a message passing mechanism, and finally the camera's extrinsic parameters (rotation and translation) are obtained.

[0138] Implement an end-to-end deep learning framework, such as Pix2Vox or PointNet++, to directly reconstruct 3D models of reference markers from multi-view images. This framework not only improves reconstruction speed but also generates more refined and accurate 3D structures. Simultaneously, reinforcement learning algorithms (such as DQN or PPO) are introduced during the construction of the 3D spatial geometric relationship network to dynamically adjust errors that occur during reconstruction. The system continuously optimizes the error correction strategy based on feedback from each adjustment, ensuring that the final 3D spatial geometric relationship network is as close to reality as possible.

[0139] To address the difficulty in accurately reconstructing the spatial geometric relationship between the target point and reference markers in traditional methods, and to further improve the accuracy and efficiency of the initial target point localization, in some embodiments, step 105 utilizes the three-dimensional spatial geometric relationship network to obtain the position information of the reference markers at the current level, as well as the relative angles and straight-line distances between the reference markers, and calculates the preliminary position of the target point, including:

[0140] Using the three-dimensional spatial geometric relationship network, the position information of the reference markers and the relative angles between the reference markers under the current observation conditions are obtained to reproduce the observation conditions when the sample under test was first observed by scanning electron microscopy. Based on the relative angle and straight-line distance of the target point relative to the reference markers recorded under the observation conditions when the sample under test was first observed by scanning electron microscopy, and combined with the position information of the reference markers and the relative angles and straight-line distances between the reference markers under the current observation conditions, the spatial geometric relationship between the target point and the reference markers is restored. Using the spatial geometric relationship, the preliminary position of the target point is calculated through an improved triangulation algorithm.

[0141] In this embodiment,

[0142] Observation conditions refer to the specific parameters recorded during the initial scanning electron microscope observation, such as magnification, viewing angle, and stage angle. These conditions are crucial for reproducing the relative position of the target point during the initial observation.

[0143] Spatial geometric relationships refer to the relative positional relationship between a target point and a reference marker, including relative angles and straight-line distances. Reconstructing this relationship is fundamental to calculating the initial position of the target point.

[0144] The improved triangulation algorithm is an optimized geometric reconstruction method that combines current observation conditions with geometric constraints from historical observation data. It accurately reconstructs the preliminary position of the target point by calculating the angular and positional differences between feature point pairs. Compared to traditional methods, the improved algorithm better handles noise, reduces error accumulation, and improves positioning accuracy.

[0145] In this embodiment, a three-dimensional spatial geometric relationship network is used to obtain the positional information of reference markers and the relative angles between reference markers under the current observation conditions. First, the observation conditions when the sample under test was first observed by scanning electron microscopy are reproduced through this network.

[0146] Then, based on the relative angle and straight-line distance between the target point and the reference marker recorded under the initial observation conditions, and combined with the position information of the reference marker under the current observation conditions and the relative angle and straight-line distance between the reference markers, the spatial geometric relationship between the target point and the reference marker is restored.

[0147] Finally, using this spatial geometric relationship and an improved triangulation algorithm, the preliminary position of the target point is calculated. This process ensures that the approximate location of the target point can be accurately found even under different observation conditions, providing a reliable basis for further precise positioning.

[0148] Here is a specific example:

[0149] Suppose an automated scanning electron microscope target point relocation system is being developed, the following techniques can be used to enhance the accuracy and efficiency of the above steps:

[0150] Using deep learning models (such as LSTM or Transformer) to analyze the observation conditions at the time of the first observation (such as magnification, viewing angle, stage angle, etc.) and predicting how to adjust under current observation conditions to most closely approximate the state of the first observation. This step helps to minimize the impact of changes in observation conditions.

[0151] Based on a three-dimensional spatial geometric relationship network, and combining the relative angles and straight-line distances of the target points relative to reference markers recorded during the initial observation, a graph neural network (GNN) model, such as GraphSAGE or GAT, is developed to automatically match and recover the spatial geometric relationships between target points and reference markers. GNNs can capture the complex interactions between feature point pairs, providing more accurate spatial relationship estimates.

[0152] Implement an end-to-end deep learning framework, such as Pix2Vox or PointNet++, to directly reconstruct 3D models of reference markers from multi-view images. Building upon this, introduce an improved triangulation algorithm that combines current observation conditions with geometric constraints from historical observation data to accurately calculate the preliminary location of the target point. For example, using a pre-trained PointNet++ model as a base, add fine-tuning layers specific to SEM images to adapt to different material and structural properties, thereby improving localization accuracy.

[0153] Throughout the process, reinforcement learning algorithms (such as DQN or PPO) are introduced to dynamically adjust errors that occur during reconstruction. The system continuously optimizes the error correction strategy based on feedback from each adjustment, ensuring that the final 3D spatial geometry is as close to reality as possible. Furthermore, self-supervised learning methods, such as contrastive learning or consistency regularization, can be used to further enhance the model's generalization ability and robustness.

[0154] To address the difficulty in accurately reproducing the initial observation conditions in traditional methods and further improve the accuracy and consistency of target point relocation, some embodiments include obtaining the position information of reference markers and the relative angles between reference markers under the current observation conditions to reproduce the observation conditions when the sample to be tested was first observed by scanning electron microscopy, including:

[0155] Based on the three-dimensional spatial geometric relationship network, the three-dimensional coordinates of each reference marker under the current observation conditions are obtained as the position information of the reference marker; according to the position information, the relative angles between all reference markers in the same layer are calculated to determine the relative angles between reference markers under the current observation conditions; the relative angles and straight-line distances between reference markers under the current observation conditions, as well as the angle of the sample stage, are adjusted to ensure that the relative angles and straight-line distances between reference markers are consistent with the originally determined relative angles and straight-line distances between reference markers during the first observation, and that the angle of the sample stage is consistent with the originally determined angle of the sample stage.

[0156] In this embodiment, relative angle and linear distance refer to the geometric relationship between reference markers within the same level, i.e., the included angle and linear distance between them. Ensuring the consistency of these parameters is fundamental to accurate relocation.

[0157] Sample stage angle adjustment refers to adjusting the angle of the platform on which the sample is placed relative to the optical axis of the scanning electron microscope to restore the observation conditions at the time of the first observation. This step ensures geometric consistency under different observation conditions.

[0158] In this embodiment of the application, in order to more accurately reproduce the initial observation conditions and further improve the accuracy and consistency of target point relocation, this scheme uses a three-dimensional spatial geometric relationship network to obtain the three-dimensional coordinates of each reference marker under the current observation conditions as its position information.

[0159] Then, based on this positional information, the relative angles between all reference markers within the same level are calculated to determine the relative angles between reference markers under the current observation conditions. Next, by adjusting the relative angles and straight-line distances between reference markers under the current observation conditions, and simultaneously adjusting the angle of the sample stage, these geometric relationships are ensured to remain consistent with those at the time of the initial observation. This process not only guarantees the consistency of the relative angles and straight-line distances between reference markers but also ensures that the angle of the sample stage is the same as that at the time of the initial observation, thus providing a reliable benchmark for subsequent precise positioning.

[0160] Here is a specific example:

[0161] Suppose an automated scanning electron microscope target point relocation system is being developed, the following techniques can be used to enhance the accuracy and efficiency of the above steps:

[0162] Deep learning models (such as PointNet++ or Pix2Vox) are used to extract and reconstruct the 3D coordinates of each reference marker from multi-view images under current observation conditions. This model can be trained on a large amount of labeled SEM image data to learn the optimal representation of different morphological features, thereby improving the accuracy of 3D coordinate extraction.

[0163] Based on the extracted 3D coordinates, a graph neural network (GNN) model, such as GraphSAGE or GAT, is developed to automatically calculate the relative angles between all reference markers within the same level. GNNs can capture the complex interactions between feature point pairs, providing more accurate spatial relationship estimates.

[0164] Reinforcement learning algorithms (such as DQN or PPO) are introduced to dynamically adjust the relative angles and straight-line distances between reference markers, as well as the angle of the sample stage. The system continuously optimizes the adjustment strategy based on feedback from each adjustment, ensuring that the final geometric relationship is as close as possible to the state at the time of the first observation. For example, self-supervised learning methods, through contrastive learning or consistency regularization, can be used to further improve the model's generalization ability and robustness.

[0165] Throughout the process, the adjusted geometric relationships and sample stage angles are monitored and recorded in real time to ensure high consistency with the data from the initial observation. Any deviations are immediately corrected to maintain high-precision repositioning. Furthermore, a closed-loop control system can be introduced to achieve real-time error correction, guaranteeing the stability and reliability of the entire repositioning process.

[0166] To address the problem of inaccurate target point localization caused by changes in sample microstructure in traditional methods, and to further improve the accuracy and stability of target point relocation, some embodiments also include:

[0167] When the sample under test is observed for the first time using a scanning electron microscope (SEM), multimodal data of the target point and its surrounding area are recorded and stored. This multimodal data includes at least spectral features and physical properties. When the sample under test is processed and then observed a second time using an SEM, the multimodal data of the target point and its surrounding area are reacquired. By comparing the multimodal data stored during the first observation with the currently acquired multimodal data, a multimodal feature matching result is generated. This multimodal feature matching result includes information on the changes in the target point and its surrounding area between the two observations. Based on this multimodal feature matching result, the target position of the target point is adjusted to ensure the accuracy of the target point's location and to prevent it from being affected by changes in the microstructure of the sample under test.

[0168] In this embodiment, multimodal data refers to various types of data about the target point and its surrounding area recorded and stored during the initial scanning electron microscope observation. These data include at least spectral features (such as energy-dispersive X-ray spectroscopy, Raman spectroscopy, etc.) and physical properties (such as surface morphology, texture, roughness, etc.). This data provides comprehensive information about the target point and its environment for subsequent feature matching and positioning adjustments.

[0169] Spectral characteristics refer to information about the composition or structure of a sample obtained through spectral analysis techniques. For example, energy-dispersive X-ray spectroscopy (EDS) can reveal the elemental composition of a sample, while Raman spectroscopy can provide information about its molecular structure. These characteristics are crucial for identifying and comparing the same location under different observation conditions.

[0170] Physical characteristics specifically refer to the morphological features of a sample surface, such as surface morphology, texture, and roughness, which are typically obtained through scanning electron microscopy (SEM) images or other microscopic imaging techniques. These properties help to accurately describe the microstructure of the sample and provide more details for feature matching.

[0171] Multimodal feature matching results are generated by comparing the multimodal data stored during the initial observation with the currently acquired multimodal data, thus producing information on the changes in the target point and its surrounding area between the two observations. This matching result not only reveals changes in microstructure but also provides reference information needed to adjust the target point's position.

[0172] In this embodiment, to more accurately address the impact of changes in the sample's microstructure on target point localization and further improve the accuracy and stability of relocation, this method records and stores multimodal data of the target point and its surrounding area during the first scanning electron microscope observation of the sample. This data includes at least spectral characteristics and physical properties. When the sample is processed and then observed a second time using a scanning electron microscope, the current multimodal data of the target point and its surrounding area are reacquired.

[0173] Then, by comparing the multimodal data stored during the first observation with the currently acquired data, a multimodal feature matching result is generated, which includes information on the changes in the target point and its surrounding area between the two observations.

[0174] Finally, based on these multimodal feature matching results, the target position of the target point is adjusted to ensure that its positioning accuracy is not affected by changes in the sample's microstructure. This process ensures high-precision target point repositioning even if the sample surface features change.

[0175] Here is a specific example:

[0176] Suppose an automated scanning electron microscope target point relocation system is being developed, the following techniques can be used to enhance the accuracy and efficiency of the above steps:

[0177] During the initial scanning electron microscope (SEM) observations, an integrated multi-sensor platform (such as combining SEM with EDS and Raman spectrometer) is used to simultaneously acquire multimodal data of the target point and its surrounding area. This data will be stored in a high-efficiency data management system, supporting rapid retrieval and comparison. For example, a database management system (DBMS) combined with cloud storage services will be used to ensure data security and accessibility.

[0178] When the sample is processed and then subjected to a second scanning electron microscope observation, the same multi-sensor platform is used again to reacquire multimodal data of the target point and its surrounding area. To ensure data consistency, standardized operating procedures and parameter configurations can be set up in the system to ensure that the observation conditions are as consistent as possible between the two observations.

[0179] Develop a deep learning model, such as a convolutional neural network (CNN) combined with a recurrent neural network (RNN), to automatically compare multimodal data stored at the time of the first observation with the currently acquired data. This model can be trained on a large amount of labeled multimodal data to learn the correlations between different features, thereby generating accurate multimodal feature matching results. For example, using a ResNet combined with an LSTM model can effectively capture spatial and temporal feature changes.

[0180] Based on the generated multimodal feature matching results, reinforcement learning algorithms (such as DQN or PPO) are introduced to dynamically adjust the position of the target point. The system continuously optimizes the adjustment strategy based on the feedback after each adjustment, ensuring that the final localization result is as close to the real situation as possible. In addition, a closed-loop control system can be introduced to achieve real-time error correction, ensuring the stability and reliability of the entire relocalization process.

[0181] Throughout the process, the adjusted geometric relationships and stage angles are monitored and recorded in real time to ensure high consistency with the data from the initial observation. Any deviations are immediately corrected to maintain high-precision repositioning. Furthermore, self-supervised learning methods can be introduced, through contrastive learning or consistency regularization, to further enhance the model's generalization ability and robustness.

[0182] This application considers that traditional triangulation algorithms often neglect the impact of observation conditions and geometric constraints (such as the fundamental matrix) on feature point matching accuracy when processing multi-view image reconstruction. Furthermore, traditional weight calculation methods fail to fully account for the influence of factors such as angle differences, position differences, and camera translation vectors on 3D coordinate reconstruction, resulting in insufficient accuracy of the reconstructed 3D coordinates in complex backgrounds or situations with significant viewpoint changes. Therefore, a new alternative scheme is proposed, which includes:

[0183] The geometric constraints include a fundamental matrix; the improved triangulation algorithm, combined with the observation conditions and geometric constraints corresponding to each sample image, utilizes the angle differences and position differences to calculate the position information of the feature point pairs in the three-dimensional coordinate system, including:

[0184] The position information of feature point pairs in the three-dimensional coordinate system is calculated using the following formula, where the position information includes three-dimensional coordinates;

[0185]

[0186] w′ i =w i ·β i

[0187] w i =exp(-α·||T) i || 2 -β·||Δp i || 2 )

[0188] β i =exp(γ·θ) i )

[0189] Where P is the 3D coordinate of the feature point, n is the number of feature point pairs, i is the index of the feature point pair, and w' i K is the combined weight of the i-th feature point pair. i x is the camera intrinsic parameter matrix corresponding to the i-th pair of feature points. 1,i and x 2,i These are the two-dimensional coordinates of the same reference marker in two different sample images, F i Let be the fundamental matrix corresponding to the i-th feature point pair, wi be the initial weight of the i-th feature point pair, and β be the initial weight of the i-th feature point pair. i α is the exponential adjustment factor for the i-th feature point pair, θ is the angular difference, Δp is the positional difference, and α is a positive adjustment parameter used to control the degree of influence of the translation vector. i It is the translation vector corresponding to the i-th pair of feature points, ||T i || represents the Euclidean norm of the translation vector, β is a positive adjustment parameter used to control the degree of influence of positional differences, and Δp i The positional difference of the i-th pair of feature points represents the relative displacement between two different observation viewpoints, ||Δp i || 2 γ is the Euclidean norm of the positional difference, γ is a positive adjustment parameter used to control the degree of influence of the rotation angle, and θ is the positional difference. i It is the angular difference of the i-th pair of feature points, representing the relative rotation angle between two different observation viewpoints.

[0190] The following is a detailed explanation of each parameter:

[0191] P: The calculated position of the feature point in the three-dimensional coordinate system. It is the final reconstructed target, representing the precise positional information of the reference marker in space. This value is calculated by a weighted average in the formula, ensuring geometric consistency under different observation conditions and improving positioning accuracy.

[0192] n: The total number of feature point pairs used for reconstruction. Each feature point pair consists of two feature points from the same physical location in images viewed from different perspectives. This number determines the richness of the dataset; more feature point pairs generally provide higher reconstruction accuracy and stability.

[0193] i: An index variable for each feature point pair, used to identify each pair of feature points. It ranges from 1 to n, iterating through all feature point pairs involved in the calculation. This index helps process the data for each feature point pair individually, ensuring that each matching pair receives appropriate weight and consideration.

[0194] w' i The combined weight of the i-th feature point pair, which incorporates the initial weight w. i and exponential adjustment factor β i The overall weight reflects the importance of each feature point pair to the final reconstruction result, ensuring that more reliable and stable feature point pairs have a greater proportion in the calculation. Its calculation formula is: w' i =w i ·β i

[0195] w i The initial weights of the i-th feature point pair, based on the translation vector T. i and positional difference Δp i The initial weights are calculated using the formula: w i =exp(-α·||T) i || 2 -β·||Δp i || 2 )

[0196] Here, α and β are positive adjustment parameters used to control the degree of influence of translation vectors and positional differences on the weights. Larger α or β values ​​will reduce the weights of feature point pairs with large translation or positional differences, thereby reducing their impact on the final result.

[0197] β i The exponential adjustment factor for the i-th feature point pair mainly reflects the angle difference θ. i The impact of feature points on reliability.

[0198] Its calculation formula is: β i =exp(γ·θ) i )

[0199] Here, γ is a positive adjustment parameter used to control the degree of influence of the rotation angle. Small angular differences will lead to β i It is close to 1, while a larger angular difference will cause β to... i The overall weight of feature point pairs can be adjusted by significantly increasing or decreasing the weight.

[0200] K i The camera intrinsic parameter matrix corresponding to the i-th feature point pair contains intrinsic parameters such as the camera's focal length and principal point position. These parameters are crucial for converting 2D image coordinates to 3D world coordinates. In the formula, Used to backproject image coordinates into three-dimensional space, Then it is used to process the fundamental matrix F i The transformation.

[0201] x 1,i and x 2,i These are the two-dimensional coordinates of the same reference marker in two different sample images. These coordinates are obtained through feature point detection and matching algorithms and are used to describe the appearance of the same physical location from different viewing angles. They are the foundation for constructing geometric relationships and the starting point for calculating three-dimensional coordinates.

[0202] F i The fundamental matrix corresponding to the i-th pair of feature points describes the geometric relationship between the two images. The fundamental matrix contains information about camera motion, including rotation and translation, used to correlate the coordinates of the same physical location from different viewing angles. In the formula, F... i It is used to calculate the relative positional relationship between pairs of feature points.

[0203] θ i : The relative rotation angle of the i-th feature point pair between two different observation viewpoints. The angle difference reflects the rotational change between the two observation viewpoints and is crucial for evaluating the reliability of the feature point pair and adjusting the overall weights. A large angle difference may lead to a large error, therefore it is necessary to use β... i Make appropriate adjustments.

[0204] Δp i : The relative displacement of the i-th feature point pair between two different observation viewpoints. The positional difference reflects the actual distance the feature points have moved between the two observation viewpoints, and is crucial for evaluating the stability of the feature point pair and adjusting the initial weights. A large positional difference may reduce the reliability of the feature point pair, therefore it is necessary to use w i Make appropriate adjustments.

[0205] α, β, and γ: Positive adjustment parameters, with α controlling the translation vector T. i For the initial weight w i The degree of influence. β control position difference Δp i For the initial weight w i The degree of influence. γ controls the rotation angle θ i For the exponential adjustment factor β i The extent of the impact.

[0206] Here is a specific example:

[0207] Suppose we are developing an automated scanning electron microscope (SEM) target point relocalization system and want to use the above formula to improve the accuracy of feature point 3D coordinate calculation. This will be implemented using the Python programming language and the OpenCV library, with specific numerical values ​​substituted into the calculations.

[0208] Parameter settings:

[0209] n: The number of feature point pairs, set to 5.

[0210] The camera intrinsic parameter matrix, assuming that all camera intrinsic parameters are the same, is simplified to the identity matrix 1.

[0211] x 1,i and x 2,i Two-dimensional coordinates of the same reference marker in two different sample images (see Table 1).

[0212] F i : The fundamental matrix, assumed to be a known value, is simplified to the identity matrix I.

[0213] α, β, γ: Adjustment parameters, set to 0.5, 0.3, 0.2 respectively.

[0214] θ i : Angular difference, assumed to be a small angular change, for example, θ1 = 0.1 radians.

[0215] Δp i Positional differences, assumed to be small displacement changes, for example, Δp1 = [2, -3] pixels.

[0216] T i Translation vector, assumed to be a small translation change, for example, T1 = [1,2,3].

[0217] Example of feature point pairs, coordinates, and parameters is shown in Table 1 below:

[0218] Table 1

[0219] i <![CDATA[x 1,i ]]> <![CDATA[x 2,i ]]> <![CDATA[T i ]]> <![CDATA[Δp i ]]> <![CDATA[θ i ]]> 1 [100,150] [105,148] [1,2,3] [2,-3] 0.1 2 [200,300] [203,297] [1,2,3] [3,-3] 0.15 3 [300,450] [305,448] [1,2,3] [5,-2] 0.2 4 [400,600] [403,597] [1,2,3] [3,-5] 0.1 5 [500,750] [505,748] [1,2,3] [5,-3] 0.15

[0220] Calculation process:

[0221] For the first pair of feature points:

[0222] w1=exp(-0.5·(1 2 +2 2 +3 2 )-0.3·(2 2 +(-3) 2 ))=exp(-0.5·14-0.3·13)

[0223] ≈0.0006

[0224] β1=exp(0.2·0.1)≈1.02

[0225] w′1=0.0006·1.02≈0.000612

[0226] Similarly, calculate the combined weight w' of other feature point pairs. i As shown in Table 2 below:

[0227] Table 2

[0228] i 1 2 3 4 5 <![CDATA[w i ]]> 0.000612 0.000558 0.000564 0.000588 0.000564

[0229] For the first pair of feature points:

[0230]

[0231] Summarize and calculate the final three-dimensional coordinates P:

[0232] Molecular part:

[0233]

[0234] Denominator:

[0235]

[0236] Final 3D coordinates P:

[0237]

[0238] P≈0.00173i+0.001983j-0.00833k

[0239] Through the above calculations, the final 3D coordinates were obtained as P≈(0.00173, 0.001983, -0.00833). These results show that even in the presence of noise, translation, and rotation, the improved triangulation algorithm can still effectively utilize feature point pairs in multi-view images to generate relatively accurate 3D coordinates. In particular, by introducing comprehensive weights and exponential adjustment factors, the emphasis on key feature points is enhanced, the influence of irrelevant feature points is reduced, and the robustness and accuracy of 3D reconstruction are improved.

[0240] This application addresses the challenge that traditional methods struggle to accurately recover the spatial geometry under initial observation conditions when relocating target points, especially when microstructural changes occur after sample processing. This leads to inaccurate initial target point location calculations, impacting the reliability of subsequent precise positioning and analysis. To address this challenge and further improve the accuracy and stability of target point relocation, a new alternative scheme is proposed, comprising:

[0241] The process of using the three-dimensional spatial geometric relationship network to obtain the position information of the reference markers at the current level, the relative angles and straight-line distances between the reference markers, and to calculate the preliminary position of the target point includes:

[0242] Determine the position information of each reference marker in the three-dimensional coordinate system under the current observation conditions.

[0243] Calculate the relative angles between reference markers and straight-line distance

[0244] The straight-line distance d of the target point relative to the reference marker, recorded under the observation conditions during the first scanning electron microscope observation of the sample under test. initial,ref Combined with the relative angles between reference markers under the current observation conditions and straight-line distance Restore the spatial geometric relationship between the target point and the reference marker, and calculate the preliminary position P of the target point using the following formula. initial :

[0245]

[0246] Where n is the number of reference markers; i is the index of the reference marker; It is the weight of the i-th reference marker, based on the distance d from the target point to the reference marker under the initial observation conditions. initial,ref The straight-line distance between the reference marker and the current observation conditions The determination, This represents the straight-line distance between the i-th reference marker and all other reference markers j. The sum; It represents the position information of the i-th reference marker in the three-dimensional coordinate system under the current observation conditions; ΔP i The displacement vector of the target point relative to the i-th reference marker is derived based on the spatial geometric relationship between the initial observation conditions and the current observation conditions. The displacement vector is calculated using the following formula:

[0247]

[0248] Where θ initial,ij It is the relative angle between the i-th and j-th reference markers under the initial observation conditions; It is the relative angle between the i-th and j-th reference markers under the current observation conditions; It is the unit vector pointing from the i-th reference marker to the j-th reference marker.

[0249] The following is a detailed explanation of each parameter:

[0250] P initial The initial position of the target point represents the calculated position of the target point relative to the reference marker under the current observation conditions. This position is derived based on the spatial geometric relationships under the initial observation conditions and the position information of the reference marker under the current observation conditions. It is a three-dimensional coordinate vector in the form of [x, y, z], where x, y, and z represent the x-coordinate, y-coordinate, and height coordinate of the target point in the three-dimensional coordinate system, respectively.

[0251] n: The number of reference markers. Reference markers are key points in a scanning electron microscope (SEM) image that can uniquely identify their location and morphology, used to construct a three-dimensional spatial geometric network. These markers provide important information about the sample structure and serve as the basis for calculating the initial location of target points. The value of n depends on the number of reference markers selected in the actual experiment; generally, more reference markers provide more accurate spatial geometric relationships.

[0252] w i The weight of the i-th reference marker measures its influence on the initial position calculation of the target point. The formula for calculating the weight is:

[0253] Here, d initial,ref It is the distance from the target point to the i-th reference marker under the initial observation conditions, while It is the sum of the straight-line distances between the i-th reference marker and all other reference markers. The weights reflect the importance of the distance between the target point and each reference marker at the time of the first observation; the closer the reference marker is, the greater its influence on the final result.

[0254] This represents the position information of the i-th reference marker in the three-dimensional coordinate system under the current observation conditions. It is a three-dimensional coordinate vector in the form [x i ,y i ,z i ] represents the specific location of the i-th reference marker under the current observation conditions. This location information is obtained through a three-dimensional spatial geometric relationship network, ensuring geometric consistency under different observation conditions.

[0255] ΔP i It is the displacement vector of the target point relative to the i-th reference marker, representing the change in direction and distance from the i-th reference marker to the target point. It is derived from the spatial geometric relationship between the initial observation conditions and the current observation conditions, and is used to correct for the target point position shift caused by changes in the sample's microstructure.

[0256] d initial,ref,i This is the straight-line distance from the target point to the i-th reference marker under the initial observation conditions. This distance is obtained from the data recorded during the initial scanning electron microscope observation and reflects the relative positional relationship between the target point and the reference marker. It is used in calculating the weight w. i and displacement vector ΔP i This plays a crucial role in ensuring the consistency of target point positioning even under different observation conditions.

[0257] This is the relative angle between the i-th and j-th reference markers under the current observation conditions. This angle describes the directional relationship between the two reference markers and is obtained through multi-view image analysis under the current observation conditions. Changes in the relative angle reflect changes in the sample's microstructure and are crucial for adjusting the target point's position.

[0258] θ initial,ij This is the relative angle between the i-th and j-th reference markers under the initial observation conditions. This angle was obtained from data recorded during the initial scanning electron microscope observation and reflects the directional relationship between the two reference markers at that time. It is the relative angle under the current observation conditions. A comparison was made to assess changes in the microstructure of the samples and to calculate the displacement vector ΔP. i .

[0259] It is the unit vector pointing from the i-th reference marker to the j-th reference marker. (Unit vector) This describes the directional relationship between two reference markers, with a magnitude of 1. The unit vector is used in calculating the displacement vector ΔP. i The direction is determined by the displacement vector, ensuring that the direction of the displacement vector correctly reflects the relative positional changes between the reference markers.

[0260] It is a unit vector The modulus, that is, its length. Because It is a unit vector, therefore its magnitude is always 1. In the formula, It is mainly used for normalization operations to ensure that the contribution ratio of each vector is consistent during the calculation. Although It is always 1, but it is retained in the formula to maintain the integrity of the mathematical expression and to facilitate understanding and expansion.

[0261] Here is a specific example:

[0262] Suppose we are developing an automated scanning electron microscope (SEM) target point relocation system and want to use the above formula to improve the accuracy of the initial target point location calculation. We will implement this using the Python programming language and the NumPy library, and perform calculations by substituting specific numerical values.

[0263] Parameter settings:

[0264] n: Number of reference markers, set to 3;

[0265]

[0266] d initial,ref The straight-line distance of the target point relative to each reference marker under the initial observation conditions is set to 100 units.

[0267] and

[0268]

[0269] θ inititul,ij :

[0270]

[0271]

[0272] Calculation process:

[0273] Calculate weight w i :

[0274]

[0275] Calculate the displacement vector ΔP i :

[0276] For i = 1:

[0277]

[0278] Similarly, calculate other ΔP i Since the angular difference is zero, all ΔP i All are zero;

[0279] Calculate the initial position P of the target point initial :

[0280]

[0281] P initial =0.107·(100,150,200)+0.136·(200,300,400)+0.098(300,450,600)

[0282] P initial =(0.107·100+0.136·200+0.098·300,0.107·150+0.136300+0.098·450,0.107·200+0.136·400+0.098·600)

[0283] P initial = (10.7+27.2+29.4,16.05+40.8+44.1,21.4+54.4+58.8)

[0284] P initial =(67.3,100.95,134.6)

[0285] The preliminary position P of the target point was obtained through the above calculations. initial ≈(67.3, 100.95, 134.6); These calculation results show that even in the presence of noise, translation, and rotation, the improved triangulation algorithm can still effectively utilize feature point pairs in multi-view images to generate relatively accurate 3D coordinates. In particular, by introducing comprehensive weights and exponential adjustment factors, the emphasis on key feature points is enhanced, the influence of irrelevant feature points is reduced, and the robustness and accuracy of 3D reconstruction are improved.

[0286] Figure 2 This application provides a schematic diagram of a target point rapid localization device (or system) based on scanning electron microscopy observation, as shown in the embodiment of this application. Figure 2 As shown, the device includes:

[0287] The construction module 21 is used to select a target point on the sample to be tested when the sample is observed by scanning electron microscope for the first time, and to establish a multi-level, multi-scale reference marker system around the target point. Each level of the reference marker system contains at least three non-collinear morphological features as reference markers to ensure that the target point selected by the sample to be tested at different magnifications has a corresponding reference marker system as a positioning reference.

[0288] The acquisition module 22 is used to adjust the angle of the sample stage on which the sample to be tested is placed, and to record the observation conditions when acquiring the sample image by a scanning electron microscope each time the angle is adjusted and the sample image containing the target point and the reference markers around the target point is acquired. The observation conditions include the angle of the sample stage and the magnification of the scanning electron microscope.

[0289] The determination module 23 is used to determine the relative angles and straight-line distances between the target points and the selected reference markers at each level, as well as between the reference markers, from each sample image;

[0290] The generation module 24 is used to determine the angular and positional differences of the same reference marker under different viewing angles based on the multiple sample images, and to reconstruct the positional information of the reference marker in the three-dimensional coordinate system based on the angular and positional differences using stereo vision technology, thereby forming a three-dimensional spatial geometric relationship network.

[0291] The calculation module 25 is used to determine the reference markers at the corresponding level according to the current magnification when the sample to be tested is processed and observed by a second scanning electron microscope, and to obtain the position information of the reference markers at the current level and the relative angles between the reference markers using the three-dimensional spatial geometric relationship network, and to calculate the preliminary position of the target point.

[0292] The determining module 23 is also used to apply a local fine-tuning algorithm, combined with the real-time acquired sample image, to move the sample stage at the sub-pixel level according to the preliminary position of the target point, until the target point is located in the center of the field of view, so as to determine the target position of the target point. During the movement of the sample stage, by monitoring and correcting possible system errors or sample displacement, it ensures that the originally determined relative angle and straight-line distance between the reference markers remain consistent.

[0293] Figure 2 The aforementioned target point rapid localization device based on scanning electron microscopy observation can perform... Figure 1 The implementation principle and technical effects of the target point rapid localization method based on scanning electron microscopy observation described in the illustrated embodiment will not be repeated here. The specific operation methods of each module and unit in the target point rapid localization device based on scanning electron microscopy observation in the above embodiments have been described in detail in the embodiments related to this method, and will not be elaborated upon here.

[0294] In one possible design, Figure 2 The target point rapid localization device based on scanning electron microscope observation in the illustrated embodiment can be implemented as a computing device, such as... Figure 3 As shown, the computing device may include a storage component 31 and a processing component 32;

[0295] The storage component 31 stores one or more computer instructions, wherein the one or more computer instructions are invoked and executed by the processing component 32.

[0296] The processing component 32 is used for the above Figure 1The embodiment describes a rapid target point localization method based on scanning electron microscopy observation.

[0297] The processing component 32 may include one or more processors to execute computer instructions to complete all or part of the steps in the above-described method. Alternatively, the processing component may be implemented as one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described method.

[0298] Storage component 31 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0299] Of course, computing devices may also include other components, such as input / output interfaces, display components, communication components, etc.

[0300] Input / output interfaces provide interfaces between processing components and peripheral interface modules, which can be output devices, input devices, etc.

[0301] The communication components are configured to facilitate wired or wireless communication between computing devices and other devices.

[0302] The computing device can be a physical device or an elastic computing host provided by a cloud computing platform. In this case, the computing device can refer to a cloud server, and the aforementioned processing components, storage components, etc., can be basic server resources rented or purchased from the cloud computing platform.

[0303] This application also provides a computer storage medium storing a computer program, which, when executed by a computer, can perform the above-described functions. Figure 1 The illustrated embodiment presents a method for rapid target point localization based on scanning electron microscopy observation.

[0304] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0305] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0306] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0307] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for quickly positioning a target point based on scanning electron microscope observation, characterized in that, include: When the sample to be tested is observed for the first time using a scanning electron microscope, a target point is selected on the sample to be tested, and a multi-level, multi-scale reference marker system is established around the target point. Each level of the reference marker system contains at least three non-collinear morphological features as reference markers to ensure that the target point selected by the sample to be tested at different magnifications has a corresponding reference marker system as a positioning benchmark. By adjusting the angle of the sample stage on which the sample to be tested is placed, each time the angle is adjusted and a sample image containing the target point and the reference markers around the target point is acquired by scanning electron microscope, the observation conditions when acquiring the sample image are recorded, including the observation angle, the sample stage angle, and the magnification of the scanning electron microscope. Determine the relative angles and straight-line distances between the target points and selected reference markers at each level, as well as between the reference markers, from each sample image; Based on the multiple sample images, the angular and positional differences of the same reference marker under different observation perspectives are determined. Then, using stereo vision technology, the positional information of the reference marker in the three-dimensional coordinate system is reconstructed based on the angular and positional differences, forming a three-dimensional spatial geometric relationship network. When the sample to be tested is processed and observed by a second scanning electron microscope, the reference markers at the corresponding level are determined according to the current magnification. The position information of the reference markers at the current level, the relative angles and straight-line distances between the reference markers are obtained using the three-dimensional spatial geometric relationship network, and the preliminary position of the target point is calculated. By applying a local fine-tuning algorithm and combining real-time acquired sample images, the sample stage is moved at the sub-pixel level according to the initial position of the target point until the target point is located in the center of the field of view, thereby determining the target position of the target point. During the movement of the sample stage, any possible systematic errors or sample displacements are monitored and corrected to ensure that the originally determined relative angles and straight-line distances between the reference markers remain consistent.

2. The method according to claim 1, characterized in that, Based on the multiple sample images, the angular and positional differences of the same reference marker under different viewing angles are determined. Then, using stereo vision technology, the positional information of the reference marker in a three-dimensional coordinate system is reconstructed based on these angular and positional differences, forming a three-dimensional spatial geometric relationship network. This includes: For each sample image, feature points of the reference marker are extracted using computer vision algorithms, and a descriptor is generated for each feature point. The feature points refer to key points in the sample image that can uniquely identify the position and shape of the reference marker. By comparing feature descriptors in different sample images, feature points of the same reference marker from different observation perspectives are identified and feature point pairs are generated. Each feature point pair consists of one or more pairs of feature points from different sample images. The feature points in each feature point pair have similar feature descriptors and correspond to the same physical location. Based on the feature point pairs, geometric constraints are applied to estimate the relative geometric relationship between different observation viewpoints, and the rotation and translation changes of the camera on the extrinsic parameters are calculated by decomposing the relative geometric relationship to obtain the camera attitude information. Based on the feature point pairs and the camera pose information, calculate the relative rotation angle of the same reference marker under different viewing angles and use it as the angle difference; calculate the relative displacement of the same reference marker under different viewing angles and use it as the position difference. By applying an improved triangulation algorithm, combined with the observation conditions and geometric constraints corresponding to each sample image, and utilizing the angular and positional differences, the positional information of the reference marker in the three-dimensional coordinate system is reconstructed, forming a three-dimensional spatial geometric relationship network.

3. The method according to claim 2, characterized in that, The improved triangulation algorithm, combined with the observation conditions and geometric constraints corresponding to each sample image, utilizes the angle and position differences to reconstruct the position information of the reference marker in the three-dimensional coordinate system, forming a three-dimensional spatial geometric relationship network, including: An improved triangulation algorithm is applied, and combined with the observation conditions and geometric constraints corresponding to each sample image, the position information of the feature point pair in the three-dimensional coordinate system is calculated using the angle difference and the position difference. By integrating the positional information of all feature point pairs, the three-dimensional coordinates of multiple feature points on each reference marker are obtained; The position information of each reference marker is determined by fitting the three-dimensional coordinates of multiple feature points of each reference marker. The position information includes three-dimensional coordinates and orientation. Based on the position information of each reference marker, a three-dimensional spatial geometric relationship network is constructed to describe the relative angles and straight-line distances between all reference markers and between reference markers and target points. In the process of constructing the three-dimensional spatial geometric relationship network, the angle differences and position differences are used to correct any possible errors.

4. The method according to claim 1, characterized in that, The process of using the three-dimensional spatial geometric relationship network to obtain the position information of the reference markers at the current level, the relative angles and straight-line distances between the reference markers, and to calculate the preliminary position of the target point includes: Using the three-dimensional spatial geometric relationship network, the position information of the reference markers and the relative angles between the reference markers under the current observation conditions are obtained, so as to reproduce the observation conditions when the sample under test was first observed by scanning electron microscopy; Based on the relative angle and straight-line distance of the target point relative to the reference marker recorded under the observation conditions when the sample under test was first observed by scanning electron microscopy, and combined with the position information of the reference marker under the current observation conditions and the relative angle and straight-line distance between the reference markers, the spatial geometric relationship between the target point and the reference marker is restored. Using the aforementioned spatial geometric relationships and an improved triangulation algorithm, the preliminary position of the target point is calculated.

5. The method according to claim 4, characterized in that, The step of obtaining the position information of the reference markers and the relative angles between the reference markers under the current observation conditions, in order to reproduce the observation conditions when the sample under test was first observed by scanning electron microscopy, includes: Based on the three-dimensional spatial geometric relationship network, the three-dimensional coordinates of each reference marker under the current observation conditions are obtained as the position information of the reference marker; Based on the location information, calculate the relative angles between all reference markers at the same level to determine the relative angles between reference markers under the current observation conditions; The relative angles and straight-line distances between reference markers and the angle of the sample stage under the current observation conditions are adjusted to ensure that the relative angles and straight-line distances between reference markers are consistent with the original relative angles and straight-line distances between reference markers during the first observation, and that the angle of the sample stage is consistent with the original determined angle of the sample stage.

6. The method according to claim 1, characterized in that, Also includes: When the sample to be tested is observed for the first time using a scanning electron microscope, multimodal data of the target point and its surrounding area are recorded and stored. The multimodal data includes at least spectral features and physical properties. When the sample to be tested is processed and subjected to a second scanning electron microscope observation, multimodal data of the current target point and surrounding area are re-acquired; By comparing the multimodal data stored during the first observation with the currently acquired multimodal data, a multimodal feature matching result is generated. The multimodal feature matching result includes information on the changes in the target point and its surrounding area between the two observations. Based on the multimodal feature matching results, the target position of the target point is adjusted to ensure the accuracy of the target point's positioning and to ensure that it is not affected by changes in the microstructure of the sample under test.

7. The method according to claim 1, characterized in that, The process of establishing a multi-level, multi-scale reference marker system around the target point, wherein each level of the reference marker system contains at least three non-collinear morphological features as reference markers, includes: establishing a three-layer reference marker system around the target point; selecting significant features of the overall structure of the test sample containing at least three non-collinear features at the macroscopic scale as reference markers in the first-layer reference marker system; selecting stable structures of the test sample containing at least three non-collinear features at the mesoscopic scale as reference markers in the second-layer reference marker system; and selecting fine features of the test sample containing at least three non-collinear features at the microscopic scale as reference markers in the third-layer reference marker system.

8. The method according to claim 3, characterized in that, The geometric constraints include the basic matrix; The improved triangulation algorithm, combined with the observation conditions and geometric constraints corresponding to each sample image, calculates the position information of the feature point pair in the three-dimensional coordinate system using the angle difference and the position difference, including: The position information of feature point pairs in the three-dimensional coordinate system is calculated using the following formula, where the position information includes three-dimensional coordinates; ； ； ； ； in, These are the three-dimensional coordinates of the feature points. It is the number of feature point pairs. It is the index of the feature point pair. It is the first The combined weight of feature point pairs, It is the first The camera intrinsic parameter matrix corresponding to each pair of feature points. and These are the two-dimensional coordinates of the same reference marker in two different sample images. It is the first The fundamental matrix corresponding to a pair of feature points. It is the first Initial weights for feature point pairs, It is the first The exponential adjustment factor for the group of feature point pairs, It's a difference in angle. It's a difference in location. It is a positive adjustment parameter used to control the degree of influence of the translation vector. It is the first The translation vector corresponding to a pair of feature points. The Euclidean norm of a translation vector. It is a positive adjustment parameter used to control the degree of influence of positional differences. It is the first The positional difference between a pair of feature points represents the relative displacement between two different observation perspectives. It is the Euclidean norm of positional difference. It is a positive adjustment parameter used to control the degree of influence of the rotation angle. It is the first The angular difference between a pair of feature points represents the relative rotation angle between two different observation perspectives.

9. The method according to claim 4, characterized in that, The process of using the three-dimensional spatial geometric relationship network to obtain the position information of the reference markers at the current level, the relative angles and straight-line distances between the reference markers, and to calculate the preliminary position of the target point includes: Determine the position information of each reference marker in the three-dimensional coordinate system under the current observation conditions. ; Calculate the relative angles between reference markers and straight-line distance ; The straight-line distance of the target point relative to the reference marker, recorded under the observation conditions during the first scanning electron microscope observation of the sample under test. Combined with the relative angles between reference markers under the current observation conditions and straight-line distance To restore the spatial geometric relationship between the target point and the reference markers, the preliminary position of the target point is calculated using the following formula. : ； in, This refers to the number of reference markers; It is an index of the reference marker; , is the The weights of the reference markers are determined based on the distances from the target point to the reference markers under the initial observation conditions. The straight-line distance between the reference marker and the current observation conditions The determination, Indicates the first One reference marker and all other reference markers The straight-line distance between The sum; It is the first under the current observation conditions Position information of each reference marker in the three-dimensional coordinate system; The target point relative to the first observation point is derived from the spatial geometric relationship between the initial observation conditions and the current observation conditions. The displacement vectors of the reference markers are calculated using the following formula: ； in This is the first observation under the conditions of the first observation. The first reference marker and the first The relative angles between the reference markers; It is the first under the current observation conditions The first reference marker and the first The relative angles between the reference markers; From the first The reference marker points to the first The unit vector of the reference marker.

10. A rapid target point localization system based on scanning electron microscopy observation, characterized in that, include: The module is used to select a target point on the sample to be tested when the sample is observed by scanning electron microscope for the first time, and to establish a multi-level, multi-scale reference marker system around the target point. Each level of the reference marker system contains at least three non-collinear morphological features as reference markers to ensure that the target point selected by the sample to be tested at different magnifications has a corresponding reference marker system as a positioning reference. The acquisition module is used to adjust the angle of the sample stage on which the sample to be tested is placed, and to record the observation conditions when acquiring the sample image by a scanning electron microscope each time the angle is adjusted and the sample image containing the target point and the reference markers around the target point is acquired, including the angle of the sample stage and the magnification of the scanning electron microscope. A determination module is used to determine, from each sample image, the relative angles and straight-line distances between the target points and selected reference markers at each level, as well as between the reference markers; The generation module is used to determine the angular and positional differences of the same reference marker under different observation angles based on the multiple sample images, and to reconstruct the positional information of the reference marker in the three-dimensional coordinate system based on the angular and positional differences using stereo vision technology, thereby forming a three-dimensional spatial geometric relationship network. The calculation module is used to determine the reference markers at the corresponding level according to the current magnification when the sample to be tested is processed and observed by a second scanning electron microscope. It uses the three-dimensional spatial geometric relationship network to obtain the position information of the reference markers at the current level and the relative angles between the reference markers, and calculates the preliminary position of the target point. The determining module is also used to apply a local fine-tuning algorithm, combined with real-time acquired sample images, to move the sample stage at the sub-pixel level according to the preliminary position of the target point, until the target point is located in the center of the field of view, so as to determine the target position of the target point. During the movement of the sample stage, by monitoring and correcting possible system errors or sample displacements, it ensures that the originally determined relative angles and straight-line distances between the reference markers remain consistent.

Images