An end-to-end intelligent manipulation method and system of a scanning electron microscope

Abstract

The application provides an end-to-end intelligent control method and system of a scanning electron microscope, and relates to the technical fields of micro-nano control and machine vision. The application converts a starting position bounding box (starting end) and a target position bounding box (end point end) into a starting heat map and a target heat map respectively, splices a current scene image, the starting heat map and the target heat map to form a multi-channel input tensor, extracts a multi-scale spatial feature map from the multi-channel input tensor through a feature extraction network, and finally inputs the multi-scale spatial feature map into a trajectory generation network, so that multi-level visual features effective for a control task can be extracted from the fused input. The sequence model characteristics can be utilized by generating the trajectory coordinate sequence in a self-recurrent manner, so that a nano-level resolution trajectory coordinate smooth, continuous and conforming to the physical motion law can be generated.

Description

Technical Field

[0001] This invention relates to the fields of micro-nano manipulation and machine vision technology, and in particular to an end-to-end intelligent manipulation method, system, storage medium and its application for a scanning electron microscope. Background Technology

[0002] Micro-nano manipulation technology is crucial in fields such as semiconductor inspection, materials science, and biomedicine. Traditional micro-nano manipulation systems typically employ a pipelined architecture of "sensing-planning-control": first, image processing algorithms (such as edge detection and template matching) are used to identify the target location and robotic arm pose from scanning electron microscope (SEM) images; then, path planning is performed; and finally, the planned path points are converted into control commands to drive the platform's movement.

[0003] However, SEM images inherently suffer from imaging artifacts, such as backscattered electron noise, charging effects (causing bright spots or stripes in the image), beam contamination (causing hydrocarbon deposits on the sample surface to blur details), and edge effects (abnormal brightening of sample edges). These artifacts severely interfere with the recognition accuracy of traditional computer vision algorithms, leading to errors in the perception module. Furthermore, the errors in this pipelined architecture accumulate and amplify at each stage: the positioning error of the perception module is passed to the planning module, and the discretization and approximation errors of the planning module are further passed to the control module, ultimately making it difficult for the overall manipulation accuracy to meet the requirements at the nanoscale (for example, even if the individual accuracy of each module is at the hundred-nanometer level, the overall system error may reach the micrometer level). At the same time, the algorithm parameters need to be readjusted for different samples (such as metal particles, organic materials, and biological tissues), resulting in poor generalization ability and difficulty in guaranteeing real-time performance.

[0004] Therefore, there is an urgent need in this field for a new, high-precision, and robust intelligent control method that can overcome the above-mentioned shortcomings and directly establish a mapping relationship between the original SEM image and the robot's motion trajectory. Summary of the Invention

[0005] This invention aims to address the technical problems in existing technologies, such as error accumulation due to the "perception-planning-control" pipeline architecture, poor robustness to SEM imaging artifacts, and difficulty in achieving high-precision control. These problems prevent the realization of high-precision, high-reliability end-to-end automated control in complex SEM noise environments. Specifically, it provides a path recognition method that can directly generate high-precision motion trajectories based on SEM images and user commands through end-to-end learning.

[0006] This invention provides an end-to-end intelligent control method for a scanning electron microscope, comprising the following steps: S100, acquiring the current scene image captured by the scanning electron microscope, and the starting position bounding box and the target position bounding box marked by the user in the current scene image; S200, converting the starting position bounding box and the target position bounding box into a starting heatmap and a target heatmap, respectively; S300, stitching the current scene image, the starting heatmap, and the target heatmap together to form a multi-channel input tensor; S400, inputting the multi-channel input tensor into a feature extraction network to extract a multi-scale spatial feature map; S500, inputting the multi-scale spatial feature map into a trajectory generation network to generate a trajectory coordinate sequence driving the movement of a micro / nano robot through an autoregressive method, the trajectory coordinate sequence being used to control the micro / nano robot to move from the starting position to the target position.

[0007] Optionally, the initial heatmap is a two-dimensional matrix with a truncated Gaussian distribution within the initial location bounding box and zero values ​​outside the region; the target heatmap is a two-dimensional matrix with a truncated Gaussian distribution within the target location bounding box and zero values ​​outside the region.

[0008] Optionally, the feature extraction network employs an encoder based on the Transformer architecture.

[0009] Optionally, the feature extraction network is a Swing Transformer encoder, which extracts multi-scale spatial feature maps through a hierarchical window attention mechanism.

[0010] Optionally, the trajectory generation network is a Transformer decoder; the generated trajectory coordinate sequence includes: based on the trajectory points predicted in the previous time step, the trajectory points at the current time step are predicted by interacting with the multi-scale spatial feature map through the cross-attention mechanism in the Transformer decoder.

[0011] Optionally, during the training phase, the feature extraction network and the trajectory generation network are jointly trained end-to-end by minimizing the loss function between the generated trajectory coordinate sequence and the real trajectory coordinate sequence.

[0012] Optionally, the trajectory coordinate sequence is sent directly to the motion controller of the micro-nano robot platform to drive the micro-nano robot to perform grasping, placement, navigation or obstacle avoidance tasks.

[0013] The present invention also provides an end-to-end intelligent control system for a scanning electron microscope, comprising: a processor and a memory; the memory for storing a computer program; and the processor for executing the computer program to implement the end-to-end intelligent control method for a scanning electron microscope as described above.

[0014] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described above.

[0015] The present invention also provides a scanning electron microscope system, comprising: a scanning electron microscope body, a micro-nano robot platform, and an end-to-end intelligent control system for the scanning electron microscope as described above; the end-to-end intelligent control system for the scanning electron microscope is communicatively connected to the scanning electron microscope body and the micro-nano robot platform, and is used to receive images acquired by the scanning electron microscope body and send control commands to the micro-nano robot platform.

[0016] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0017] This invention effectively avoids the error propagation and accumulation problems between multiple independent modules in the traditional "perception-planning-control" pipeline by deeply integrating visual perception and motion control. The invention defines the basic input and constraints of the manipulation task by converting the starting position bounding box (starting end) and the target position bounding box (ending end) into a starting heatmap and a target heatmap, respectively. Then, by stitching the current scene image, the starting heatmap, and the target heatmap into a multi-channel input tensor, discrete position information is converted into a continuous feature representation rich in spatial context information, facilitating neural network fusion and learning. Next, a feature extraction network extracts multi-scale spatial feature maps from the multi-channel input tensor, fusing visual and task information along the channel dimension to provide comprehensive input for the feature extraction network. Finally, by inputting the multi-scale spatial feature maps into a trajectory generation network, multi-level visual features effective for the manipulation task can be extracted from the fused input. The trajectory coordinate sequence generated through an autoregressive approach can utilize the characteristics of sequence models to generate smooth, continuous, and physically motion-compliant nanometer-resolution trajectory coordinates. This invention generates multi-scale spatial feature maps in a single neural network and optimizes global parameters using an autoregressive generation method. This avoids the stepwise propagation of perception, planning, and control errors in traditional methods, enabling end-to-end vision-driven direct control. It prevents errors from being amplified step by step and significantly improves control accuracy, robustness, and success rate in complex SEM noise environments. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the end-to-end intelligent control method for a scanning electron microscope. Detailed Implementation

[0019] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0020] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the embodiments of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0021] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims. In the description of this application, it should be understood that the terms "first," "second," "third," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0022] Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0023] Example 1

[0024] In this embodiment, as Figure 1 The end-to-end intelligent control method for a scanning electron microscope, as shown, includes the following steps:

[0025] S100: Acquire the current scene image acquired by the scanning electron microscope, as well as the starting position bounding box and the target position bounding box marked by the user in the current scene image;

[0026] S200: Convert the starting position bounding box and the target position bounding box into a starting heatmap and a target heatmap, respectively;

[0027] S300: The current scene image, the initial heatmap, and the target heatmap are stitched together to form a multi-channel input tensor;

[0028] S400: Input the multi-channel input tensor into the feature extraction network to extract multi-scale spatial feature maps;

[0029] S500: Input multi-scale spatial feature maps into the trajectory generation network, and generate trajectory coordinate sequences that drive the movement of micro-nano robots through an autoregressive method. The trajectory coordinate sequences are used to control the movement of micro-nano robots from the starting position to the target position.

[0030] In this embodiment, the initial heatmap is a two-dimensional matrix with a truncated Gaussian distribution within the initial location bounding box and zero values ​​outside the region; the target heatmap is a two-dimensional matrix with a truncated Gaussian distribution within the target location bounding box and zero values ​​outside the region. This transforms the coordinates of the discrete initial and target location bounding boxes into a continuous probability distribution, which not only strengthens the importance of the target center region (consistent with the attention mechanism) but also avoids interference from irrelevant regions through truncation. This provides the feature extraction network with richer and easier-to-learn positional constraint information, thereby improving the model's convergence speed and the final positional regression accuracy, while also avoiding the feature sparsity problem of traditional one-hot encoding.

[0031] Specifically, the Gaussian heatmap generation method for the initial and target heatmaps is based on the theory of truncated Gaussian distribution, and realizes the probabilistic representation of the bounding box (initial position bounding box or target position bounding box) through the following mathematical model:

[0032] For a given bounding box b=[x min y min x max y max First, calculate its center coordinates (c). x c y ) and dimensional parameters (w, h):

[0033] Formula (1);

[0034] Formula (2);

[0035] Then define the standard deviation. An adaptive function for the maximum side length of the bounding box:

[0036] Formula (3);

[0037] Here, α is an adjustable scaling factor, and after testing, when α∈[0.3, 0.5], it can highlight the importance of the center point, ensure the truncation characteristics of the distribution, and provide a certain degree of positional uncertainty for the feature extraction network to learn.

[0038] The value of heatmap (initial heatmap or target heatmap) h(x, y) is calculated by the following piecewise function:

[0039] Formula (4);

[0040] The above function ensures that the heatmap reaches a maximum value of 1 at the center of the corresponding bounding box, decays radially according to a Gaussian distribution to the boundary, and has a value of 0 in the region outside the boundary.

[0041] Compared to traditional binary masking or direct coordinate concatenation methods, the spatial continuity of the truncated Gaussian heatmap allows the feature extraction network to effectively learn location information (starting or target position) through gradient descent; the center response enhancement mechanism aligns with the human visual attention mechanism; and the truncation characteristic avoids global information interference and reduces the redundancy of the feature space.

[0042] In this embodiment, the feature extraction network employs a Transformer-based encoder. Specifically, the feature extraction network is a Swing Transformer encoder, which extracts multi-scale spatial feature maps through a hierarchical window attention mechanism. Utilizing the Swing Transformer's hierarchical window attention mechanism, it can efficiently capture both local detail features and global contextual dependencies of an image simultaneously, enabling a better understanding of complex micro / nano scenes and long-range spatial relationships (such as the relative positions of starting points, target points, and obstacles), thus laying a solid feature foundation for generating reasonable and smooth trajectories.

[0043] Specifically, the Swin Transformer encoder employs a hierarchical window attention mechanism to achieve efficient fusion of local and global features through the following mathematical framework:

[0044] The input image X∈R^(h×w×3) is divided into non-overlapping M×M pixel blocks (patches), resulting in a patch sequence x∈R^(N×C). The patches are then mapped to a high-dimensional feature space through a linear embedding layer.

[0045] Formula (5);

[0046] Where N = (h / M) × (w / M) is the total number of patches, and M is the patch size set manually, such as 16, 32, etc.

[0047] X ∈ R^(h×w×3) means that the input is a color image (RGB image);

[0048] h represents the image height, in pixels;

[0049] w represents the image width, measured in pixels;

[0050] 3 represents the number of channels, corresponding to the three color channels: red (R), green (G), and blue (B).

[0051] x∈R^(N×C) arranges all pixel blocks into a sequence in a certain order (usually from left to right and from top to bottom raster scan order), and each pixel block is flattened and used as a token.

[0052] C=3M 2 C represents the patch dimension, and represents the vector dimension of each patch after it has been flattened.

[0053] w e ∈R^(3M²×D), w e Embedded weight matrix, where D is the dimension of the hidden layer.

[0054] In the L-th layer Swing Transformer block, self-attention is first calculated within the local window:

[0055] Formula (6);

[0056] in,

[0057] Q represents the query matrix, and Formula (7);

[0058] K represents the bond matrix, and Formula (8);

[0059] V represents the value matrix, and Formula (9);

[0060] dk represents the scaling factor.

[0061] In this embodiment, to achieve cross-window information interaction, a shifted window strategy is adopted: odd-numbered layers use regular window partitioning, while even-numbered layers shift the window by M / 2 pixels horizontally and vertically. The output features of layer l... Calculated using the following residual join:

[0062] Formula (10);

[0063] Formula (11);

[0064] Here, w-MSA represents window multi-head self-attention, and MLP (Multi-Layer Perceptron) is a two-layer feedforward network. Through L layers stacked, the final output is a multi-scale feature map. ∈R^(N×D).

[0065] In this embodiment, the trajectory generation network is a Transformer decoder. Generating a trajectory coordinate sequence includes: based on the trajectory points predicted at the previous time step, interacting with multi-scale spatial feature maps through the cross-attention mechanism in the Transformer decoder to predict the trajectory points at the current time step. Simultaneously, the autoregressive approach naturally models the temporal dependencies between trajectory points, ensuring that the generated trajectory sequence is smooth and continuous; while the cross-attention mechanism enables the decoding process to dynamically focus on the image region most relevant to the current predicted point, achieving fine-grained interaction between trajectory generation and visual perception, further improving the accuracy and reasonableness of the trajectory.

[0066] Specifically, the Transformer decoder, based on a sequence-to-sequence learning paradigm, generates trajectory points sequentially through the following recursive mechanism:

[0067] The decoder maintains a sequence of query vectors. , where T is the trajectory length. At time t, the update of the query vector qt follows the following rules:

[0068] Formula (12);

[0069] in,

[0070] e start This indicates a learnable starting token embedding;

[0071] Embed(·) represents the coordinate embedding function. These are the predicted coordinates from the previous moment.

[0072] In this embodiment, the coordinate embedding function Embed(·) is implemented by a linear layer with a weight matrix w. embed ∈R^(2×D), where D is the dimension of the hidden layer of the model. That is, the two-dimensional coordinate vector [x, y] is multiplied by matrix [x, y]. w embed Projected into D-dimensional space.

[0073] Meanwhile, to avoid error accumulation during the sequential generation of trajectory points using a recursive mechanism, a closed-loop control strategy can be adopted during real-time control. That is, instead of generating the complete trajectory sequence all at once, a rolling horizon approach is used.

[0074] After each k-step trajectory prediction, the current SEM scene image is reacquired, and the current actual position is used as the new starting point to re-perform target detection and trajectory prediction, thereby continuously correcting errors and ensuring the accuracy of the operation.

[0075] The query vector first captures the temporal pattern of the generated trajectory through a self-attention layer:

[0076] Formula (13);

[0077] Then, spatial context information is extracted by interacting with the encoder feature map F through a cross-attention layer:

[0078] Formula (14);

[0079] The formula for calculating cross attention is:

[0080] Formula (15);

[0081] Finally, the two-dimensional coordinates are regressed through a two-layer fully connected network:

[0082] Formula (16);

[0083] Where w1∈R^(D×512) and w2∈R^(512×2) are weight matrices. The above autoregressive method makes the prediction of each trajectory point depend on historical information, naturally satisfying the constraints of trajectory smoothness and continuity.

[0084] In this embodiment, the feature extraction network and the trajectory generation network are jointly trained end-to-end by minimizing the loss function between the generated trajectory coordinate sequence and the real trajectory coordinate sequence.

[0085] Specifically, the loss function used for end-to-end joint training is designed based on robust regression theory, using L1 distance instead of L2 distance to reduce the impact of outliers.

[0086] Given the true trajectory ;

[0087] and predicted trajectory Position loss L pos Defined as:

[0088] Formula (17);

[0089] Compared to L2 loss:

[0090] Formula (18);

[0091] The L1 loss penalty for large biases increases linearly rather than quadratically, preventing the training process from being dominated by a few outliers. To further improve the smoothness of the trajectory, a velocity regularization term between adjacent points is introduced:

[0092] Formula (19);

[0093] Where Δt is the sampling time interval. Total loss function The weighted sum of the two items:

[0094] Formula (20);

[0095] Where λ=0.1 is the balance coefficient.

[0096] In this embodiment, the gradient of the loss function is calculated using the backpropagation algorithm:

[0097] Formula (21);

[0098] Where θ represents all learnable parameters of the trajectory generation network, and the parameters are updated using the Adam w optimizer:

[0099] Formula (22);

[0100] Where η is the learning rate, m t and v t First-order and second-order moment estimates, respectively, λ w This represents the weight decay coefficient. The learning rate is dynamically adjusted using a cosine annealing strategy.

[0101] Formula (23);

[0102] Where η max =1×10 -4 η min =1×10 -6 T max This represents the total number of training rounds.

[0103] In this embodiment, the trajectory coordinate sequence is directly sent to the motion controller of the micro-nano robot platform to drive the micro-nano robot to perform grasping, placement, navigation or obstacle avoidance tasks.

[0104] Example 2

[0105] This embodiment provides an image enhancement strategy based on the previous embodiment.

[0106] In this embodiment, the following enhancement strategies were designed to improve the robustness of the model based on the special imaging mechanism of scanning electron microscope images.

[0107] For the input image I∈R^(h×w), a random geometric transformation is first applied:

[0108] Formula (24);

[0109] Where T rot (θ) represents a rotational transformation, θ∈(-15°, 15°); T flip (p) represents the flip transformation, p=0.5.

[0110] Then, a photometric transformation is applied to simulate imaging artifacts:

[0111]

[0112] Where γ~U(0.8, 1.2) simulates brightness variation, β~U(-0.1, 0.1) simulates bias drift, and N(0, σ²) is Gaussian noise simulating backscattered electron noise. The standard deviation σ of the Gaussian noise is determined by calculating the average standard deviation of pixel values ​​in the background region of multiple static SEM images.

[0113] To simulate the charging effect, Gaussian bright spots were added at random locations:

[0114] Formula (25);

[0115] in,

[0116] K represents the number of Gaussian bright spots, and K∈(0,3);

[0117] A k This represents the amplitude of the Gaussian bright spot, and A k ∈(0.3, 0.7);

[0118] r k Let r represent the radius of the Gaussian spot. k The radius is ∈(10, 30), and the range of values ​​for the amplitude and radius of the Gaussian bright spot is obtained by analyzing the statistical characteristics of the Gaussian bright spot in the SEM image when the charging effect occurs.

[0119] The aforementioned enhancement operations (random geometric transformation, photometric transformation, and Gaussian bright spot) are randomly combined during training, enabling the feature extraction network to learn an invariant representation of scanning electron microscope imaging artifacts.

[0120] Example 3

[0121] Based on any of the above embodiments, this embodiment provides an end-to-end intelligent control system for a scanning electron microscope.

[0122] In this embodiment, the system includes a processor and a memory; the memory is used to store a computer program; the processor is used to execute the computer program to implement the end-to-end intelligent control method for the scanning electron microscope as described above.

[0123] In this embodiment, the processor includes at least a spatiotemporal registration data acquisition module, a location information encoding module, a multi-scale spatial feature extraction module, an autoregressive trajectory generation module, an end-to-end optimization training module, and an intelligent control execution module.

[0124] In this embodiment, the spatiotemporal registration data acquisition module is configured to simultaneously acquire image sequences and robot motion trajectories within the scanning electron microscope cavity, thereby constructing the training dataset required for end-to-end learning.

[0125] Specifically, the spatiotemporal registration data acquisition module includes:

[0126] The multi-sample acquisition unit targets interdisciplinary micro- and nano-feature samples such as ternary cathode material spheres, carbon fibers, metal-organic frameworks, CPU bumps, and graphene. The multi-sample acquisition unit executes predefined manipulation trajectories in a high-vacuum environment. These trajectories are automatically generated by an autoregressive trajectory generation module, including randomly sampled linear motion, circular motion, and serpentine scanning trajectories around the target sample region within the workspace. These trajectories are repeatedly executed under different initial poses and motion parameters to cover the main motion patterns and effective manipulation areas that may occur during micromanipulation.

[0127] The timestamp synchronization unit achieves time alignment between scanning electron microscope image frames and the three-dimensional pose data of the nano-positioning stage through a hardware triggering mechanism. Specifically, the frame scanning synchronization output signal of the scanning electron microscope is used as an external trigger source. When each frame image is scanned, the motion controller is triggered to synchronously record the high-precision encoder reading of the nano-positioning stage, and the timestamp is uniformly allocated by the main control module, thereby achieving millisecond-level synchronization accuracy.

[0128] In one possible implementation, timestamp synchronization is achieved in the following way:

[0129] The frame sync out signal provided by the SEM is used as an external interrupt trigger source, directly connected to the digital input port of the motion controller. Upon receiving the rising edge trigger signal, the motion controller immediately latches the current three-dimensional coordinates of the nanostation encoder and sends these coordinates along with the trigger timestamp to the host computer. The host computer software matches this timestamp with the acquisition completion time of the corresponding SEM image frame, thereby achieving millisecond-level synchronization accuracy.

[0130] The data annotation unit is used to establish an image-trajectory-boundary box triplet dataset. The target location bounding box is manually annotated in the SEM image, while the starting location bounding box is determined by the calibration pose of the probe tip. Specifically, the probe used for micro-manipulation is first moved to a known reference marker within the field of view for geometric calibration. Then, during data acquisition, the nanostation encoder acquires the spatial position of the probe tip in real time and projects it onto the synchronously acquired SEM image according to the calibration relationship, thereby achieving reliable annotation of the probe tip's starting position.

[0131] In this embodiment, the location information encoding module is configured to convert discrete bounding box coordinates into a continuous spatial probability distribution, thereby achieving effective fusion of location information and image features.

[0132] In this embodiment, the location information encoding module specifically includes a Gaussian heatmap generation unit, a multi-channel stitching unit, and a normalization processing unit.

[0133] Specifically, the Gaussian heatmap generation unit is used to convert the bounding box into a truncated Gaussian distribution heatmap, avoiding the feature sparsity problem of traditional one-hot encoding. The multi-channel stitching unit is used to stitch the initial heatmap h... s Target heatmap h g The image is stitched with the original scanning electron microscope image (original scene image) I to form a three-channel input tensor. The normalization unit is used to map all channel values ​​to the range [0, 1] to eliminate dimensional differences between different channels.

[0134] In this embodiment, the multi-scale spatial feature extraction module is configured to extract hierarchical visual feature representations from the fused multi-channel input tensor.

[0135] In this embodiment, the multi-scale spatial feature extraction module specifically includes a convolutional encoder backbone network, a feature pyramid unit, and a feature compression unit.

[0136] Specifically, the convolutional encoder backbone network adopts a ResNet, Vision Transformer, or SwinTransformer architecture, extracting multi-scale spatial feature maps through multi-layer convolutions or self-attention mechanisms. Feature pyramid units are used to construct feature layers at different resolutions, taking into account both local details and global context. Feature compression units reduce feature dimensions through 1×1 convolutions, reducing the computational burden on subsequent decoders.

[0137] In this embodiment, the autoregressive trajectory generation module is configured to generate a sequence of control path points based on the order obtained by extracting multi-scale spatial feature maps.

[0138] In this embodiment, the autoregressive trajectory generation module specifically includes a Transformer decoder unit, a cross-attention fusion unit, and a coordinate regression unit.

[0139] Specifically, the Transformer decoder unit models the temporal dependencies between trajectory points through a self-attention mechanism. The cross-attention fusion unit interacts with the decoder query vector and the encoder feature map to extract the spatial context related to the current trajectory point. The coordinate regression unit maps the decoder output to two-dimensional coordinates (x, y) through a fully connected layer, achieving sub-pixel accuracy prediction.

[0140] In this embodiment, the end-to-end optimization training module is used to optimize the parameters of the entire feature extraction network and trajectory generation network through the backpropagation algorithm, so as to achieve the globally optimal mapping from image to trajectory.

[0141] In this embodiment, the end-to-end optimization training module specifically includes a loss function calculation unit, a data augmentation unit, and an optimizer scheduling unit.

[0142] Specifically, the loss function calculation unit uses L1 distance loss to supervise trajectory prediction accuracy. The data augmentation unit applies enhancement operations such as random rotation, flipping, brightness adjustment, and Gaussian noise to the scanning electron microscope images. The optimizer scheduling unit uses the Adamw optimizer combined with a cosine annealing learning rate strategy to achieve stable convergence.

[0143] In this embodiment, the intelligent control execution module is configured to deploy a trained feature extraction network to a real-time control system, driving the micro-nano robot to complete complex control tasks. The intelligent control execution module includes:

[0144] Real-time inference unit for efficient network forward propagation on a single graphics processor;

[0145] The trajectory smoothing unit is used to perform spline interpolation on the predicted discrete path points to generate a continuous executable trajectory.

[0146] The closed-loop feedback unit is used to continuously acquire scanning electron microscope images and dynamically adjust the subsequent trajectory according to the current state.

[0147] Example 4

[0148] This embodiment provides a computer-readable storage medium based on any of the above embodiments.

[0149] The computer-readable storage medium in this embodiment stores a computer program, which, when executed by a processor, implements the end-to-end intelligent control method for the scanning electron microscope as described above.

[0150] Example 5

[0151] This embodiment provides a scanning electron microscope system based on any of the above embodiments.

[0152] In this embodiment, the scanning electron microscope system includes a scanning electron microscope body, a micro-nano robotic platform, and an end-to-end intelligent control system for the scanning electron microscope as described above.

[0153] In this embodiment, the end-to-end intelligent control system of the scanning electron microscope is communicatively connected to the scanning electron microscope body and the micro-nano robot platform, and is used to receive images acquired by the scanning electron microscope body and send control commands to the micro-nano robot platform.

[0154] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. An end-to-end intelligent control method for a scanning electron microscope, characterized in that, Includes the following steps: S100: Acquire the current scene image acquired by the scanning electron microscope, as well as the starting position bounding box and the target position bounding box marked by the user in the current scene image; S200, The starting position bounding box and the target position bounding box are respectively converted into a starting heatmap and a target heatmap for characterizing the position probability distribution; the starting heatmap is a two-dimensional matrix that has a truncated Gaussian distribution within the starting position bounding box and a value of zero outside the region; The target heatmap is a two-dimensional matrix that has a truncated Gaussian distribution within the target location bounding box and a value of zero outside the region; S300: The current scene image, the initial heatmap, and the target heatmap are stitched together to form a multi-channel input tensor; S400: Input the multi-channel input tensor into the feature extraction network to extract a multi-scale spatial feature map; S500: Input the multi-scale spatial feature map into the trajectory generation network, and generate a trajectory coordinate sequence that drives the movement of the micro-nano robot through an autoregressive method. The trajectory coordinate sequence is used to control the micro-nano robot to move from the starting position to the target position. The trajectory generation network is a Transformer decoder; Generating the trajectory coordinate sequence includes: based on the trajectory points predicted at the previous time step, interacting with the multi-scale spatial feature map through the cross-attention mechanism in the Transformer decoder to predict the trajectory points at the current time step.

2. The end-to-end intelligent control method for a scanning electron microscope according to claim 1, characterized in that, The feature extraction network employs an encoder based on the Transformer architecture.

3. The end-to-end intelligent control method for a scanning electron microscope according to claim 2, characterized in that, The feature extraction network is a Swing Transformer encoder, which extracts multi-scale spatial feature maps through a hierarchical window attention mechanism.

4. The end-to-end intelligent control method for a scanning electron microscope according to claim 1, characterized in that, The feature extraction network and the trajectory generation network are jointly trained end-to-end by minimizing the loss function between the generated trajectory coordinate sequence and the real trajectory coordinate sequence.

5. The end-to-end intelligent control method for a scanning electron microscope according to claim 1, characterized in that, The trajectory coordinate sequence is directly sent to the motion controller of the micro-nano robot platform to drive the micro-nano robot to perform grasping, placement, navigation or obstacle avoidance tasks.

6. An end-to-end intelligent control system for a scanning electron microscope, characterized in that, include: Processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program to implement the end-to-end intelligent control method for a scanning electron microscope as described in any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the end-to-end intelligent control method for the scanning electron microscope as described in any one of claims 1 to 5.

8. A scanning electron microscope system, characterized in that, include: The scanning electron microscope body, the micro-nano robotic platform, and the end-to-end intelligent control system for the scanning electron microscope as described in claim 6; The end-to-end intelligent control system of the scanning electron microscope is communicatively connected to the scanning electron microscope body and the micro-nano robot platform, and is used to receive images acquired by the scanning electron microscope body and send control commands to the micro-nano robot platform.

Images