Electron microscope imaging target accurate movement control method based on visual servo

Abstract

The invention discloses an electron microscope imaging target accurate movement control method based on visual servo, and belongs to the technical field of electron microscopic imaging and movement control. Comprising the following steps: constructing a target data set in a low-magnification scene, training a fine-tuning target tracking model, and outputting a target center point; performing real-time communication with the electron microscope to obtain low-power image frames, identifying each target by using the ideal model, generating an online target tracking task sequence, and continuously outputting a tracking frame of a current target individual in sequence; according to the pixel error, mapping a calibration coefficient related to multiplying power into a displacement reference quantity to execute PID closed loop, and driving the sample table to move until the target individual meets the center; and switching to high magnification to complete observation and returning to low magnification to complete multi-target observation one by one. According to the method, re-search and re-calibration after rate switching are avoided through the process of'retention-rollback recovery-relative displacement inheritance ', the robustness is improved in combination with thread decoupling and adaptive parameters, multi-target rapid, accurate and stable high-power observation can be achieved in a millisecond-level control period, and the method is suitable for large-scale popularization and application. The switching time delay is obviously shortened, the invalid range and the manual burden are reduced, and the experiment verifies that the method has effectiveness and superiority.

Description

Technical Field

[0001] This invention relates to the field of electron microscopy imaging and motion control technology, specifically to a technical solution for using visual servoing and closed-loop control to precisely move multiple pre-selected target areas sequentially to the center of the field of view for high-magnification observation in a transmission electron microscope. Background Technology

[0002] Electron microscopes, as high-resolution microscopic imaging instruments capable of observing micro- and nano-sized samples in real space, play an irreplaceable role in physics, chemistry, materials science, biology, and other fields. However, transmission electron microscopes (TEM) currently cannot achieve full automation, largely due to multi-source uncertainties and multi-coordinate coupling issues in precise target movement and centering. These uncertainties primarily stem from: 1. The mapping between imaging coordinates, physical stage coordinates, and image shift coil control coordinates changes with magnification and aberration correction states; 2. Pixel-to-nanometer / current calibration exhibits scale-dependent and nonlinear characteristics; 3. Magnification switching is accompanied by lens hysteresis and field-of-view shift, and the sample stage suffers from backlash, creep, and thermal drift; 4. The field of view shrinks significantly at high magnification, making it extremely sensitive to centering errors at low magnification; 5. Under low-dose imaging or low-contrast sample conditions, increased measurement noise leads to unstable target centroid estimation. Furthermore, achieving unmanned automated control of electron microscopes typically requires a closed-loop control cycle within tens to hundreds of milliseconds, further exacerbating the difficulty of precise movement and stable centering.

[0003] Existing methods often rely on manual positioning and centering at low magnification, resulting in low efficiency and poor consistency. Some automated solutions often implement open-loop or semi-closed-loop centering control for single targets, lacking overall process design for multi-target scenarios. A common practice is to re-search or rebuild the template after magnification switching, failing to effectively inherit the target bounding box and relative position information obtained in the low-magnification stage. This leads to repeated re-acquisition and recalibration between each high-low magnification switch. At the same time, some solutions rely solely on sample stage movement while ignoring the image shift channel or failing to coordinate the two, which can easily introduce overshoot and oscillation under nonlinear and saturation constraints, resulting in a double loss of efficiency and accuracy. Summary of the Invention

[0004] To address the shortcomings of the existing technologies, the purpose of this invention is to: establish a target bounding box set and record relative displacement at low magnification; independently acquire data in real-time and write it to a single-frame buffer, while the tracking thread outputs the target center and confidence level; the control thread uses the "target center - field of view center" error calibrated by magnification to enter a PID closed loop, enabling rapid target centering. Once the target is centered, the low-magnification tracking box is frozen as an anchor point, high-magnification acquisition is switched, and the data is rolled back to restore the target; the initial bounding box of the next target is calculated based on the current low-magnification position and relative displacement, and this process is repeated, achieving rapid, accurate, and stable high-magnification observation of multiple targets. The technical solution includes the following steps:

[0005] The technical solution adopted by the present invention to achieve the above objectives is as follows:

[0006] A method for precise movement control of an electron microscope imaging target based on vision servoing, comprising:

[0007] A dataset was created by acquiring electron microscope images in low-magnification scenarios. The network structure parameters were optimized through iterative training based on a neural network model to obtain an ideal model, which was used to identify the center point and confidence level of target individuals in the output electron microscope images in low-contrast and weak-texture scenarios.

[0008] It communicates with the electron microscope in real time to acquire low-magnification image frames to be identified, calculates field-of-view parameters, uses an ideal model to identify each target in the field of view of the current frame, generates an online target tracking task sequence, and continuously outputs the tracking box, center point and confidence of the current target individual in the low-magnification scene in sequence.

[0009] Based on the pixel error between the target center and the field of view center, the displacement of the sample stage is mapped according to the magnification, and PID closed-loop control is executed to move the sample stage so that the target individual meets the center in the field of view.

[0010] Freeze the current low-magnification tracking box as a reserved box, switch to the high-magnification scene to acquire electron microscope images of the target individual and record the magnification and pixel size data to achieve high-magnification observation, and then fall back to the low-magnification scene and check and restore the fallback deviation;

[0011] Based on the low-magnification center of the current target individual and its relative displacement, the initial center and initial bounding box of the next target are calculated, the online target tracking task sequence is optimized, and the above process is iteratively executed with the interaction command of the displacement stage to realize electron microscopy observation of each target individual under low-magnification field of view, closed-loop moving displacement stage, and high-magnification field of view until all targets in the task sequence are observed.

[0012] The target individual is a microscopic cell under an electron microscope.

[0013] The data set creation process includes: acquiring electron microscope images, performing data augmentation, image segmentation and labeling, and dividing the dataset.

[0014] The data enhancements include, but are not limited to, one or more of the following: contrast / brightness perturbation, mild noise and blurring, subpixel translation / scaling, local occlusion, and compression artifacts; and the execution order is not limited to the above order.

[0015] The labeling is used to mark the pixel outline and geometric center of each target individual in the electron microscope imaging image;

[0016] The dataset is divided proportionally into a training set, a validation set, and a test set. The training set is used for training, the validation set is used to verify the model's accuracy, and the test set is used for testing.

[0017] The neural network model adopts one of the following, including but not limited to Stark and MixFormer; the iterative training and optimization of network structure parameters based on the neural network model to obtain the ideal model includes: training with AdamW and cosine annealing learning rate, using the region of interest IoU, center error and success rate as monitoring indicators to stop early, and obtaining an online ideal model that stably outputs "target center point + confidence" during runtime.

[0018] The field of view parameters include: any two target individuals relative pixel distance Current center coordinates ;

[0019] The identification of each target in the current frame's field of view involves identifying the outer contour box and geometric center;

[0020] The online target tracking task sequence is optimized based on spatial proximity and stage travel constraints. Spatial proximity means that the individual closest to the center of the field of view is searched as the first target, starting from the current sample stage position, and the shortest path connecting each target individual is formed according to the Euclidean distance to constitute the tracking task sequence. Stage travel constraints are set based on the movement range of the two-dimensional displacement stage.

[0021] It also includes removing "unreliable" centroids based on a confidence threshold, and processing the centroids... Apply a light smoothing / or extrapolation process.

[0022] The pixel error of the "target center - field of view center" , where t represents the t-th tracking;

[0023] The sample stage displacement ;

[0024] The execution of the PID closed loop is as follows: the two PIDs respectively accumulate the axial position reference based on the displacement reference increment of each axis, compare it with the axial position feedback to obtain the error, and then calculate the control quantity according to the position type PID and apply it to the corresponding axis.

[0025] The criteria for determining the center position are that the L2 norm of the pixel error is not greater than a preset threshold and the continuous holding time is not less than a preset duration.

[0026] It also includes: according to a fixed control cycle Calculate the discrete PID control quantities of the two axes to drive the XY axis motion of the sample stage, so that the error converges quickly and meets the "centering" criterion;

[0027] in,

[0028] ;

[0029] ;

[0030] in These represent the proportional, integral, and derivative coefficients of the PID controller, respectively; when And duration It was determined to be the heart at that time.

[0031] The backoff deviation verification and recovery includes: after restoring the retained box, obtaining the backoff deviation compensation vector by image matching, and performing short closed-loop correction until the recovery threshold is met. The short closed-loop correction is to use a smaller movement step for scaling back.

[0032] The initial center and initial bounding box of the next target are calculated based on the low-magnification center of the current target individual and its relative displacement, as follows: based on the relative distance between the center points of two targets i and j. The center point of the current target i Calculate the center point of the next target The pixel position, i.e. .

[0033] When observing each target individual in an online target tracking task sequence, if the confidence level is below the threshold for multiple consecutive frames or recapture fails, recapture is performed based on relative displacement and historical trajectory; if it still fails, it reverts to the previous stable state or the reserved frame is reinitialized and the task progress is restored.

[0034] The present invention has the following beneficial effects and advantages:

[0035] 1. Continuous Inheritance and Rapid Switching: Before each transition from low to high magnification, the current low-magnification tracking frame is fixed as the position anchor point. After returning to low magnification from the end of high-magnification observation, this anchor point is restored first, and based on the initially established relative positional relationships between targets, the initial tracking frame of the next target is directly generated and put into closed-loop control. The above-mentioned "preservation-restoration-relative position inheritance" mechanism avoids the re-search and recalibration process after switching, reducing the time consumption of stage travel and magnification trial and error. Combined with optimization of spatial proximity and stage travel constraints for the access order, it can significantly shorten the switching latency between targets, reduce invalid movement and mechanical wear, and improve batch processing throughput.

[0036] 2. High Precision and Steady-State Convergence: Based on real-time tracking, the deviation between the center of the tracking frame and the center of the field of view is acquired as an error signal. PID closed-loop control is used to directly drive the sample stage movement, enabling the position error to converge rapidly and remain stable within a small tolerance. This closed-loop scheme automatically compensates for the influence of mechanical factors such as stage backlash, creep, and micro-vibration on positioning through error feedback, thereby achieving precise centering of the target within the high-magnification field of view and avoiding positioning deviations caused by various mechanical errors.

[0037] 3. Robustness and Availability: When short-term occlusion, contrast changes, or sudden increases in drift cause tracking instability, the system can use relative position references and historical trajectories to generate candidate regions for recapture. If consecutive recapture attempts fail, the system automatically reverts to the previous stable state or retains the bounding box for reinitialization, while preserving task progress and key parameters for rapid recovery. These mechanisms enable long-term, multi-target tasks to continue even under disturbances, significantly reducing reliance on operator experience and frequent manual intervention, thus improving overall availability and operational stability. Attached Figure Description

[0038] Figure 1 This is a flowchart of the method of the present invention.

[0039] Figure 2 This is the flowchart of the tracking and PID closed-loop control module.

[0040] Figure 3 This is a schematic diagram of the test results under 4K magnification of the actual TEM machine.

[0041] Figure 4 This is a schematic diagram of the test results under a 30K magnification on a real TEM machine. Detailed Implementation

[0042] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0044] like Figure 1-2 As shown, the specific solution of the present invention is as follows:

[0045] S1: Construct a tracking dataset and fine-tune the target tracking model to achieve stable tracking of TEM images, including the following steps.

[0046] S11: Data Acquisition: Acquire image sequences containing target motion and slight drift under low magnification, stable beam and constant sampling parameters; save the original resolution.

[0047] S12: Align the target with the rectangle on the axis to obtain the target bounding box. ,in This refers to the coordinates of the geometric center of the target. This refers to the width of the target bounding box. This refers to the height of the target bounding box.

[0048] S13: Data Augmentation: Contrast / brightness perturbation, mild Gaussian noise and blur, subpixel translation / scaling, local occlusion, and compression artifact simulation, with the magnitude controlled within the range that does not change the discernibility of the target.

[0049] S14: Model Fine-tuning: Select a general target tracking model; set the input size and search region ratio, and fine-tune using the AdamW optimizer and cosine annealing learning rate; use IoU, center error, and success rate as monitoring indicators, and stop early to prevent overfitting. General target tracking models can include Stark, MixFormer, etc.

[0050] S2: Construct a high-speed image acquisition channel and obtain the initial target and its relative position.

[0051] S21: A separate image acquisition thread is set up to acquire electron microscope images in real time via Socket communication. The images are placed in a message queue, from which other programs can read the images.

[0052] S22: Load the set of target bounding boxes for the region to be observed. ,in This represents the i-th target bounding box. The area to be observed can be manually selected through the front-end interactive interface or automatically provided by the target detection algorithm. The pixel center point of each target is calculated. ,in Represents the pixel coordinates of the center point of the i-th bounding box, and the coordinates of the current view center. ;

[0053] S23: Calculate and store the distance between the center points of each target pixel in the pixel coordinate system of the same low-magnification field of view. And the size information of the target bounding box, as a reference for subsequent initialization.

[0054] S3: Real-time target tracking

[0055] S31: Thread decoupling: Establish a tracking thread and a control thread, which share a read-only structure to pass the latest tracking results, including the pixel coordinates of the target center point and the confidence score.

[0056] S32: Preprocessing and Inference: Input the target bounding boxes of the current frame and the i-th target into the target tracking model for inference, and output the target center. and rating , where t represents the t-th tracking.

[0057] S33: Result Smoothing: Based on Scoring Filter the target center point, when When the value is below a threshold, it is marked as "unreliable," and... Lightweight smoothing / extrapolation is employed to avoid target boxes with excessively low confidence levels interfering with subsequent entry into the control loop.

[0058] S4: PID closed-loop control of the sample stage to achieve centering.

[0059] S41: Error Calculation: The control thread reads the latest target center point position. Calculate the position error at pixel coordinates ,in These represent pixel errors in the x and y directions, respectively.

[0060] S42: Pixel-to-Stage Command Step Unit Mapping: Using Magnification-Related Calibration Factor Pixel errors are mapped to sample stage translation commands;

[0061] .

[0062] S43: PID update: based on a fixed control cycle Execute discrete PID:

[0063] ;

[0064] ;

[0065] in These represent the proportional, integral, and derivative coefficients of the PID controller, respectively. Directly driving the X and Y motion of the sample stage allows for rapid error convergence; when... And duration When determining "reaching the center", the center point of the i-th target is... .

[0066] S5: High-magnification observation and retraction after the target moves to the center of the field of view.

[0067] S51: Freeze Preserved Box: Once the target moves to the center of the field of view, tracking updates are immediately paused, and the current low-magnification tracking box is frozen. Record the center point and the size of the target box.

[0068] S52: Magnification Switching and Acquisition: Sends a magnification switching command to the TEM to high magnification, performs high-magnification observation / acquisition, and records metadata (magnification, pixel size, etc.).

[0069] S53: Rollback and Verification: After completing the observation, switch back to low magnification and restore the tracking frame. The backtracking deviation is calculated using an image matching algorithm. If the deviation after backtracking is... Then, a short closed-loop correction is performed until the recovery threshold is met again.

[0070] S6: Determine the next objective and continue closed-loop control.

[0071] S61: Initial bounding box estimation for the next target j: based on the relative distance between the center points of the two targets i and j. The center point of the current target i Calculate the center point of the next target The pixel position, i.e. .

[0072] S62: Execute S32 to send the current frame and initial target bounding box into the tracking module and execute subsequent steps.

[0073] like Figure 3 The image shown is a schematic diagram of the test results at 4K magnification on a real TEM. Figure 4 The diagram shows the test results at 30K magnification on a real TEM machine. Under 4K / 30K conditions, the system first activates a separate image acquisition thread to acquire electron microscope images in real time. Then, the target bounding box of the area to be observed is loaded and input into the tracking network along with the electron microscope image. The tracking thread is then activated to acquire the current target position in real time. The error between this position and the center of the field of view is calculated and input into a PID controller to obtain the movement distance of the electron microscope stage. After the stage moves, the tracker updates the target's current position, and this process is repeated until the error converges. After error convergence, the magnification is increased for detailed observation. After observation, the magnification is restored to its initial value, and image matching is used for correction. After correction, the target bounding box of the next target is calculated based on the relative position information between targets, and the above process is repeated.

[0074] The above description is merely a preferred embodiment of the present invention and does not constitute any limitation on the present invention. Any simple modifications, alterations, or equivalent structural changes made to the above embodiments based on the technical essence of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for precise movement control of an electron microscope imaging target based on visual servoing, characterized in that, The application relates to a method for realizing electron microscope observation under low magnification, high magnification and low magnification. The method comprises the following steps: Collecting electron microscope imaging images to make a data set under low magnification, iteratively training a neural network model to optimize network structure parameters to obtain an ideal model, and using the ideal model to identify the center point and confidence of a target individual in an electron microscope imaging image under a low-contrast and weak-texture scene of the electron microscope; Real-time communication with the electron microscope end to obtain a low-magnification image frame to be identified, calculate a field of view parameter, identify each target in the current frame field of view by using the ideal model, generate an online target tracking task sequence, and continuously output a tracking frame, a center point and a confidence of a current target individual under low magnification in sequence; Based on the pixel error of the target center and the field of view center, the magnification is calibrated and mapped into a sample stage displacement amount, PID closed-loop control is performed on the sample stage movement, so that the target individual in the field of view meets the centering requirement; Freezing the current low-magnification tracking frame as a reserved frame, switching to a high-magnification scene to collect target individual electron microscope images and recording magnification and pixel size data to realize high-magnification observation, then returning to the low-magnification scene and correcting and recovering the return deviation; 2.The visual servoing-based control method for precise movement of an electron microscope imaging target according to claim 1, wherein, According to the low-magnification center of the current target individual and the relative displacement thereof, the initial center and the initial frame of the next target are calculated, the online target tracking task sequence is optimized, and the above process is iteratively executed by interacting with the displacement stage to realize electron microscope observation under the low-magnification field of view-closed-loop movement of the displacement stage-high-magnification field of view for each target individual, until the observation of all targets in the task sequence is completed. The target individual is a microscopic cell under the electron microscope perspective; The method comprises the following steps: The data set is made by collecting electron microscope imaging images, performing data enhancement, image segmentation labeling and data set division operations; The data enhancement includes but is not limited to one or more of contrast / luminance disturbance, slight noise and blur, sub-pixel translation / scaling, local occlusion and compression artifacts, and the execution sequence is not limited to the above sequence; 3.The visual servoing-based control method for precise movement of an electron microscope imaging target according to claim 1, wherein, The labeling is to draw the pixel outer contour frame and geometric center of each target individual in the electron microscope imaging image; 4. The method of claim 1, wherein the method is a visual servoing based control method for precise movement of a target in an electron microscope. The field of view parameters include: any two target individuals relative pixel distance Current center coordinates ; The data set is divided into a training set, a verification set and a test set in proportion, the training set is used for training, the verification set is used for verifying the model precision, and the test set is used for testing. The neural network model adopts one of Stark and MixFormer, and the ideal model is obtained by iteratively training the neural network model to optimize network structure parameters, using AdamW and a cosine annealing learning rate, early stopping according to the monitoring indexes of an interesting region IoU, a center error and a success rate, and obtaining an online ideal model which stably outputs "target center point + confidence" in running time. The identification of each target in the current frame field of view is to identify the outer contour frame and the geometric center; The online target tracking task sequence is optimized according to spatial proximity and stage range constraints; the spatial proximity is to search the nearest individual to the current sample stage position as the first target, link the shortest path of each target individual according to the Euclidean distance, and form a tracking task sequence; and the stage range constraint is to set a constraint condition according to the two-dimensional displacement stage movement range.

5. The method of claim 1, wherein, Also included is the deletion of "unreliable" center points according to a confidence threshold, and the center points A light smoothing / or extrapolation process is taken.

6. The method of claim 1, wherein, the pixel error of the "target center - visual field center" where t denotes the tth tracking; The sample stage displacement amount ; The PID closed loop is formed by two paths of PID, each of which accumulates the displacement reference increment of each axis to form an axial position reference, compares the axial position feedback with the error to calculate the control amount according to the position type PID and apply it to the corresponding axis. The criterion for reaching the heart is that the two norm of the pixel error is not greater than a preset threshold and the continuous retention time is not less than a preset time length.

7. The method of claim 6, wherein, Further comprising: Fixed control cycle The two-axis discrete PID control amount is calculated, the XY axes of the sample table are driven to move, the error is quickly converged, and the "centering" criterion is met. Wherein, ； ； wherein respectively represent the proportional coefficient, the integral coefficient and the derivative coefficient of the PID controller; when and the duration is determined as a heart.

8. The method of claim 1, wherein, The check and recovery of the rollback deviation includes: obtaining a rollback deviation compensation vector by image matching after the reserved frame is recovered, and performing a short closed loop correction until a recovery threshold is met, wherein the short closed loop correction is a scaling rollback using a smaller moving step.

9. The method of claim 1, wherein the method further comprises: The initial center and initial bounding box of the next target are calculated based on the low-magnification center of the current target individual and its relative displacement, as follows: based on the relative distance between the center points of two targets i and j. The center point of the current target i Calculate the center point of the next target The pixel position, i.e. .

10. The method of claim 1-9, wherein, When performing an online target tracking task sequence to observe each target individual one by one, if the confidence of consecutive multiple frames is lower than a threshold or the recapture fails, recapture is performed based on the relative displacement and the historical trajectory; if it still fails, it is rolled back to the last stable state or the reserved frame is reinitialized and the task progress is recovered.

Images