Stm image surface species tracking method and system based on visual large model

Abstract

The first aspect of the technical scheme of the present application discloses a scanning tunneling microscope image surface species tracking method based on a visual large model. The second aspect of the present application discloses an automatic tracking system for implementing the above scanning tunneling microscope image surface species tracking method, comprising an image input module, a large model reasoning module, a data analysis module and a result output module. Compared with the prior art, the present application has the beneficial effects of high automation and intelligence, high precision and quantification, strong robustness, excellent practicability and the like.

Description

Technical Field

[0001] This invention relates to a method and system for automated analysis of continuous image sequences acquired by scanning tunneling microscope (STM) using artificial intelligence technology. Specifically, it relates to a method and system for tracking species such as atoms / molecules on the surface of scanning tunneling microscope images based on a large visual model, belonging to the field of image processing and scientific instrument data analysis technology. Background Technology

[0002] As one of the core technologies of scanning probe microscopy (SPM), STM enables in-situ real-time observation of dynamic processes on surfaces and interfaces at atomic to nanoscale resolution, and has become the most direct means of studying surface physicochemical phenomena such as catalytic reactions and thin film growth. By continuously and rapidly scanning the same area, time-series image data recording the movement, configuration evolution, and other behaviors of surface species (such as adsorbed atoms, molecular clusters, and nanoparticles) can be obtained.

[0003] However, the analysis of such continuous STM image sequences has long faced severe challenges. The primary challenge is image drift: due to factors such as thermal drift and piezoelectric ceramic creep, complex global distortions (such as translation, rotation, and scaling) occur between consecutive frames, severely disrupting the spatiotemporal correspondence. Although the applicant has separately filed a patent application for an automated high-precision drift correction method based on Scale Invariant Feature Transform (SIFT) and Random Sample Consensus (RANSAC) algorithms (application number 202511520124.7, application date October 23, 2025, invention title "An Automated Processing Method and System for Continuous Scanning Probe Microscope Images"), which effectively solves the image registration problem, the second challenge that follows—how to automatically, accurately, and quantitatively extract dynamic information from the massive amount of corrected image data—becomes particularly prominent.

[0004] Currently, tracking surface species largely relies on researchers manually selecting frames frame by frame or using segmentation methods based on fixed thresholds. These methods are not only inefficient and subjective, but also unable to effectively handle complex scenarios such as morphological changes, contrast fluctuations, and the appearance, disappearance, and adhesion of objects during movement. In recent years, large-scale visual models (such as the Segment Anything Model, SAM, and its second-generation model SAM2) have made breakthroughs in the field of general image segmentation. However, when directly applied to scientific research data such as STM images, which have unique contrast mechanisms (tunneling current) and noise characteristics, there are adaptation challenges. More importantly, scientific instruments such as STMs are usually deployed as edge devices, and their built-in computing units generally lack the support of high-performance graphics processing units (GPUs). Large-scale models such as SAM2 have huge computational requirements and are difficult to run in real time on resource-constrained edge devices, which greatly limits the application of advanced artificial intelligence technologies in practical scientific research scenarios.

[0005] Therefore, there is an urgent need in this field for an automated dynamic tracking solution that can deeply integrate cutting-edge AI capabilities, balance high precision and high efficiency, and can be deployed in actual STM device computing environments. Summary of the Invention

[0006] The purpose of this invention is to solve the problems that existing technologies cannot perform automated, intelligent, and quantitative tracking and analysis of surface species in continuous STM images, and that existing large models are difficult to deploy and apply on STM edge devices.

[0007] To achieve the above objectives, a first aspect of the present invention discloses a method for surface species tracking in scanning tunneling microscope images based on a large visual model, characterized by comprising the following steps:

[0008] S1. Data Acquisition Steps: Acquire a sequence of drift-corrected, continuous scanning tunneling microscope images; S2. Model segmentation and tracking steps: The continuous scanning tunneling microscope image sequence is input into the visual basic model. Based on the initial interactive prompts for the target species provided by the user in the first frame or other specified frames, the visual basic model generates the pixel-level mask segmentation result of the target species in the current frame. Then, by utilizing the video object segmentation capability of the visual basic model, the identification and mask information of the target species are automatically propagated to all subsequent frames of the current frame in the continuous scanning tunneling microscope image sequence, realizing fully automatic cross-frame tracking. S3. Trajectory and parameter calculation steps: Based on the mask of each target species in each frame output by the large visual model, quantitative analysis is performed to generate the motion trajectory of each target species and calculate its kinematic and / or morphological parameters.

[0009] Preferably, the drift correction in step S1 is achieved by a method including the following steps: extracting feature points of adjacent frames in the continuous scanning tunneling microscope image sequence and matching them; using a robust estimation algorithm to screen matching points and estimate the geometric transformation model for image correction; and registering the images according to the geometric transformation model to eliminate inter-frame translation, rotation, scaling and shearing distortions.

[0010] Preferably, the basic visual model in step S2 is the Segment Anything Model 2 model, or a derivative model of the Segment Anything Model 2 model that has been optimized with lightweight features.

[0011] Preferably, the derived model employs network optimization techniques such as depthwise separable convolution to reduce computational load and memory consumption, thereby adapting to the computational capabilities of edge scanning tunneling microscope devices, such as the EdgeTAM model.

[0012] Preferably, the initial interactive prompt in step S2 is a dot prompt or a box prompt.

[0013] Preferably, in step S2, during the process of automatically propagating the target species' identifier and mask information to all subsequent frames, the segmentation result is corrected by adding new interactive prompts to specific frames in which there is a deviation between the target species' mask and its true outline. The corrected information is then propagated.

[0014] Preferably, in step S3, the kinematic parameters include at least one of the following: center of mass coordinates, instantaneous velocity, or cumulative displacement.

[0015] Preferably, in step S3, the morphological parameters include mask area and / or average pixel intensity value.

[0016] Preferably, the average pixel intensity value is calculated as follows: after converting the original scanning tunneling microscope image into a grayscale image, the average grayscale value of all pixels within the target species mask area is calculated.

[0017] A second aspect of the present invention discloses an automated tracking system for implementing the above-described method for tracking species on the surface of scanning tunneling microscope images, characterized in that the automated tracking system comprises: Image input module for loading drift-corrected sequential scanning tunneling microscope image sequences; The large model inference module is used to load the basic visual model, receive interactive prompts, and perform masking segmentation and video tracking of the target species for each frame in a continuous scanning tunneling microscope image sequence. The data analysis module is used to calculate the motion trajectory, kinematic parameters, and morphological parameters of the target species based on the masking results obtained from the large model inference module. The results output module is used to visualize the tracking results obtained by the data analysis module and export quantitative data.

[0018] Compared with the prior art, the present invention has the following beneficial effects: 1. High automation and intelligence: Through the "one-time annotation, full-process tracking" capability of the visual large model, the entire process of dynamic analysis of surface species is automated, which greatly improves data processing efficiency and reduces manual operation costs and professional technical thresholds. 2. High precision and quantification: Thanks to the pixel-level segmentation accuracy of the large model, it can output multi-dimensional and high-precision quantitative data such as trajectory, velocity, area, and signal strength, providing a reliable basis for scientific research; 3. Strong robustness: It can intelligently handle complex situations such as target deformation, brightness changes and partial occlusion, overcoming the limitations of traditional methods in such scenarios; 4. Excellent practicality: By introducing lightweight models (such as EdgeTAM), advanced algorithms can be adapted to the computing power limitations of edge devices such as STM, solving the key problem of the difficulty in deploying high-performance AI models in actual scientific research scenarios, and promoting the deep integration of artificial intelligence and cutting-edge scientific instruments. Attached Figure Description

[0019] Figure 1 This is a flowchart of the present invention; Figure 2 This is a system block diagram of the present invention; Figure 3 This is a schematic diagram of continuous STM image sequence sampling after drift correction in a specific embodiment; Figure 4 To and Figure 3 Corresponding to the sequence, after processing by the method described in this invention, a visualization diagram of the target species segmentation and cross-frame tracking results is shown. In the diagram, the object outlines (mask boundaries) with different colors or numbers output by the model are superimposed to demonstrate the segmentation and tracking effects. Figure 5 Based on Figure 4 The tracking data automatically generates a schematic diagram of the surface species' movement trajectory. In the diagram, the background can be one frame of an STM image, on which multiple curves (trajectory lines) of different colors are superimposed. Each curve connects the position of the same target in different frames, clearly showing the movement trajectory. Detailed Implementation

[0020] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0021] Taking the tracking of the dynamic behavior of Pt atoms / clusters on FeOx crystal planes in a CO atmosphere as an example, the method disclosed in this invention is further illustrated. The application of the technical solution provided by this invention to study the surface dynamic evolution of Pt single-atom catalysts during low-temperature CO oxidation reactions specifically includes the following steps: 1. Data preparation and drift correction A set of 100 consecutively acquired STM images, named 00.jpg to 99.jpg, were obtained from an STM experiment under a CO atmosphere and within a specific temperature range. Using the method described in the applicant's drift correction patent (application number 202511520124.7, application date October 23, 2025, titled "An Automated Processing Method and System for Continuous Scanning Probe Microscope Images"), the image sequence was highly registered to eliminate inter-frame misalignment caused by thermal drift, resulting in an aligned image sequence. Figure 3 As shown, the image sequence sampled frames after drift correction demonstrate the dynamic evolution of nanoclusters on the sample surface during the catalytic reaction process.

[0022] 2. Model Loading and Initialization On a workstation with GPU computing power, load a pre-trained SAM2 model, such as the sam2.1_hiera_large version based on the Hiera architecture. Note that if you need to deploy the SAM2 model locally on an STM device, you can choose its lightweight version—the EdgeTAM model. Initialize the video tracking state of the loaded model and load the corrected image sequence paths.

[0023] 3. Initial labeling and segmentation In the first frame of the image sequence (00.jpg), clicks are made at the center of 10 Pt atoms / clusters to be tracked via a programmatic interface (such as the add_click_annotations function) to provide point cues. Based on these cues, the SAM2 model generates precise pixel-level masks for these 10 targets on the fly and assigns them unique identifiers, for example, IDs: 1 to 10.

[0024] 4. Automatic video tracking By invoking the video propagation function of the SAM2 model (such as propagate_in_video), the SAM2 model will automatically track these 10 targets in subsequent frames 1 to 99 (01.jpg to 99.jpg) and output the mask corresponding to each target in each frame. Figure 4 As shown, after processing by Visual Large Model (SAM2), target clusters in each frame are automatically identified and segmented, and their pixel-level mask contours (distinguished by different colors or identifiers) are precisely annotated on the image.

[0025] 5. Interactive correction (optional) During automatic tracking, discrepancies were observed between the masks of target ID 9 in frame 8 (09.jpg), target ID 3 in frame 12 (13.jpg), and target ID 6 and their actual contours. Click prompts were then added to the corresponding frames for manual correction. The SAM2 model immediately updated the masks based on the new prompts and seamlessly propagated the corrected results to all subsequent frames, ensuring tracking accuracy.

[0026] 6. Data Quantification and Analysis Iterate through all processed frames and perform post-processing calculations on the mask of each target in each frame, including: Centroid coordinates: The center position (X,Y) of the object is obtained by finding the positions of non-zero pixels in the mask and calculating their average value. Mask area: The sum of the pixels in the binarized mask gives the pixel area occupied by the object. Average pixel intensity: The Z-Channel information of the original STM image is converted into a grayscale image. The pixel values ​​of the target area are extracted using a mask and the average value is calculated. This average value can reflect the electronic state or height information of the object.

[0027] 7. Results Output and Visualization The calculation results (centroid coordinates, mask area, average pixel intensity) for each object in each frame are stored in a data structure (such as a Pandas DataFrame). Finally: The motion trajectory (the line connecting the centers of mass) is overlaid on the image sequence to generate a visual tracking video or image, such as... Figure 5 As shown, the system automatically calculated and plotted the centroid movement trajectories of each target species, visually presenting the migration paths and behavioral patterns of different clusters during the observation period. Export the quantified data as a CSV file to facilitate subsequent kinetic analysis.

[0028] Results: By analyzing the derived data, the migration rate, diffusion coefficient, aggregation growth (area increase) or decomposition (area decrease) kinetics of each Pt species, as well as changes in their apparent contrast, can be precisely quantified, providing direct experimental evidence for a deeper understanding of the dynamic behavior of active sites in the CO oxidation reaction.

Claims

1. A method for surface species tracking in scanning tunneling microscope images based on a large visual model, characterized in that, Includes the following steps: S1. Data Acquisition Steps: Acquire a sequence of drift-corrected, continuous scanning tunneling microscope images; S2. Model segmentation and tracking steps: The continuous scanning tunneling microscope image sequence is input into the visual basic model. Based on the initial interactive prompts for the target species provided by the user in the first frame or other specified frames, the visual basic model generates the pixel-level mask segmentation result of the target species in the current frame. Then, by utilizing the video object segmentation capability of the visual basic model, the identification and mask information of the target species are automatically propagated to all subsequent frames of the current frame in the continuous scanning tunneling microscope image sequence, realizing fully automatic cross-frame tracking. S3. Trajectory and parameter calculation steps: Based on the mask of each target species in each frame output by the large visual model, quantitative analysis is performed to generate the motion trajectory of each target species and calculate its kinematic and / or morphological parameters.

2. The method for surface species tracking in scanning tunneling microscope images based on a large visual model as described in claim 1, characterized in that, The drift correction in step S1 is achieved by a method including the following steps: extracting feature points of adjacent frames in the continuous scanning tunneling microscope image sequence and matching them; using a robust estimation algorithm to screen matching points and estimate the geometric transformation model for image correction; and registering the images according to the geometric transformation model to eliminate inter-frame translation, rotation, scaling and shearing distortion.

3. The method for surface species tracking in scanning tunneling microscope images based on a large visual model as described in claim 1, characterized in that, The basic visual model mentioned in step S2 is the Segment Anything Model 2 model, or a derivative model of the Segment Anything Model 2 model that has been optimized with lightweight features.

4. The method for surface species tracking in scanning tunneling microscope images based on a large visual model as described in claim 3, characterized in that, The derived model employs depthwise separable convolution technology to reduce computational load and memory usage, thereby adapting to the computational capabilities of edge scanning tunneling microscope equipment.

5. The method for surface species tracking in scanning tunneling microscope images based on a large visual model as described in claim 1, characterized in that, The initial interactive prompt in step S2 is either a dot prompt or a box prompt.

6. The method for surface species tracking in scanning tunneling microscope images based on a large visual model as described in claim 1, characterized in that, In step S2, during the process of automatically propagating the target species' identifier and mask information to all subsequent frames, the segmentation result is corrected by adding new interactive prompts to specific frames in which there is a deviation between the target species' mask and the true outline. The corrected information is then propagated.

7. The method for surface species tracking in scanning tunneling microscope images based on a large visual model as described in claim 1, characterized in that, In step S3, the kinematic parameters include at least one of the following: center of mass coordinates, instantaneous velocity, or cumulative displacement.

8. The method for surface species tracking in scanning tunneling microscope images based on a large visual model as described in claim 1, characterized in that, In step S3, the morphological parameters include mask area and / or average pixel intensity value.

9. The method for surface species tracking in scanning tunneling microscope images based on a large visual model as described in claim 8, characterized in that, The average pixel intensity value is calculated as follows: after converting the original scanning tunneling microscope image into a grayscale image, the average grayscale value of all pixels within the target species mask area is calculated.

10. An automated tracking system for implementing the surface species tracking method for scanning tunneling microscope images according to any one of claims 1 to 9, characterized in that, The automated tracking system includes: Image input module for loading drift-corrected sequential scanning tunneling microscope image sequences; The large model inference module is used to load the basic visual model, receive interactive prompts, and perform masking segmentation and video tracking of the target species for each frame in a continuous scanning tunneling microscope image sequence. The data analysis module is used to calculate the motion trajectory, kinematic parameters, and morphological parameters of the target species based on the masking results obtained from the large model inference module. The results output module is used to visualize the tracking results obtained by the data analysis module and export quantitative data.

Images