An agent system for multimodal gui interaction understanding and automatic operation for sem

Abstract

The application relates to an intelligent agent system for multi-modal GUI interaction understanding and automatic operation of SEM, which dynamically analyzes the GUI interface state, operation instruction and task target through multi-modal perception and semantic understanding, and introduces a communication and state feedback mechanism between intelligent agents, so that after completing physical operations such as cabin door closing, the device operation intelligent agent can transmit the operation result and state information to the GUI intelligent agent, and the GUI intelligent agent continues to execute the subsequent software operation process accordingly. Compared with the prior art, the application does not rely on a predefined interface or a fixed script, but realizes the orderly connection and adaptive control of the SEM software layer and the device layer operation through multi-agent cooperation and cross-layer communication, and improves the automation degree, stability and universality of the complex operation process.

Description

Technical Field

[0001] This invention relates to the field of automation control, and more particularly to an intelligent agent system for multimodal GUI interactive understanding and automatic operation of SEM. Background Technology

[0002] As an important characterization instrument in materials science, semiconductor manufacturing, and nanotechnology, the operation of a scanning electron microscope (SEM) involves multiple complex steps, including parameter configuration, imaging mode switching, sample positioning, and image acquisition. Currently, mainstream SEM systems primarily rely on manual operation through a graphical user interface, which is highly dependent on operator experience. This results in high learning costs, low operational efficiency, poor repeatability, and difficulties in automation and standardization, especially in high-throughput acquisition and long-term unattended scenarios.

[0003] While existing technologies can achieve streamlined operations such as "automatic movement—automatic imaging—automatic identification—automatic analysis" or "image—spectrum—phase" to a certain extent, and can automate some tasks through vendor APIs or scripts, they still exhibit a strong "task-object binding" characteristic overall. Dedicated software typically relies on pre-defined object models, threshold rules, and fixed analysis chains, limiting its adaptability to variations in sample morphology, background complexity, and imaging conditions. Once the application expands from specific statistical tasks like particle / mineral phase analysis to more general combined processes such as "feature finding—decision amplification—mode switching—dynamic parameter tuning—review and verification," it often requires redevelopment or recalibration, making reuse and migration difficult. Furthermore, existing solutions generally lack semantic understanding and stable, operable mapping capabilities for SEM software GUIs. Most systems are centered around internal modules or predefined button flows, making it difficult to uniformly perceive and determine the state of multimodal information. Therefore, they lack sufficient coverage of dynamic GUI scenarios such as pop-up interruptions, parameter linkage, mode switching, plugin window collaboration, and differences in interface between different software versions, resulting in limited robustness of automated operations in real continuous experimental environments. Furthermore, automation frameworks based on vendor API scripts suffer from engineering deficiencies due to their strong dependence on interfaces: the capabilities of APIs from different versions and brands are inconsistent, and some critical operations exist only in the GUI without being made publicly available; interface upgrades, permission changes, or plugin updates can easily cause script failures, resulting in poor system portability and maintainability across devices and versions. Moreover, these frameworks often rely primarily on manual workflow orchestration, lacking the ability to automatically generate executable operation sequences from instructions through task decomposition, state tracking, and action planning, making it difficult to achieve adaptive closed-loop parameter tuning based on real-time image quality and instrument status. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide an intelligent agent system for multimodal GUI interactive understanding and automatic operation in SEM, thus solving the deficiencies of the prior art.

[0005] The objective of this invention is achieved through the following technical solution: an intelligent agent system for multimodal GUI interactive understanding and automatic operation in SEM, the system comprising a GUI intelligent agent set on the software side and a robotic arm intelligent agent set on the device side;

[0006] The GUI agent is configured to receive a start signal, send a sample change command to the robotic arm agent, and start the SEM software automation after receiving the robotic arm agent's ready signal. At the same time, it calls the pre-trained region selection model, selects candidate observation regions on the sample, performs focusing and image quality optimization operations in the candidate regions and acquires preview images, inputs the preview images into the VLM model for target feature determination, takes a picture and saves the image when the target features are met, and sends a success command to the robotic arm agent as a completion mark.

[0007] The robotic arm agent is configured to receive signals sent by the GUI agent through the VLA model on the robotic arm, and output corresponding parameters to control the robotic arm to complete the corresponding operation before sending a ready signal back to the GUI agent.

[0008] The GUI agent uses a VLM model to jointly model the SEM preview image, SEM software interface screenshot, and structured parameters with the task target text, thereby achieving a closed loop of target feature recognition, structured result output, and driving subsequent automatic operations.

[0009] The GUI agent uses a VLM model to jointly model SEM preview images, SEM software interface screenshots, structured parameters, and task target text, including:

[0010] Collect the current SEM preview image and related metadata, and write the set words into the prompt words;

[0011] The input image is preprocessed, and then the image and text are input into the VLM model to obtain a structured output that includes at least whether the target is hit, confidence score, target feature text description, and next action suggestion;

[0012] The output is parsed and verified, and the GUI atomic operation sequence is executed accordingly.

[0013] When the system determines that the target has not been hit, it updates the field of view, magnification, and focus parameters based on the action suggestions and re-captures the preview image, forming an iterative closed loop.

[0014] When the hit threshold and image quality threshold are met, the image saving process begins, and the system returns to the initialization state to complete an automatic sampling task.

[0015] The process of inputting images and text into the VLM model to obtain a structured output that includes at least whether the target was hit, confidence level, target feature text description, and next action suggestion includes:

[0016] Image and text are input together into the VLM model. Visual-language fusion is expressed through a two-layer multilayer perceptron (MLP). Let the local 2×2 image patch features output by the visual encoder of the VLM model be... spliced ​​together The fused visual marker is ,in, For learnable projection matrices, For bias terms, It is a non-linear activation function. Represents local topographic information of the image, i=1, 2, 3, 4;

[0017] For a given layer of input data H containing text and visual markers, the formula is used to... By assigning attention weights to different visual markers based on the task text, image evidence can be retrieved according to target conditions, and a decision can be made. These represent the query vector, key vector, and value vector, respectively. For learnable parameters, For each attention head, the feature dimensions, This is the scaling factor;

[0018] The VLM model output is generated through autoregression. At step t, the language model's probability for the next output visual token is... By constraining the output format in the prompts, the generated results are stably parsed into a structured output that includes whether the target was hit, confidence level, target feature text description, and next action suggestion. For multimodal input, In hidden state, This is for outputting the projection matrix.

[0019] The VLA model is based on Model building is used to map multimodal observation information and high-level operational semantics into continuously executable device control actions;

[0020] The model includes a multimodal input encoding module, a visual-language backbone network, an action expert module, and an action sequence output module;

[0021] The multimodal input encoding module is used to receive and encode image data from the device's vision sensor, language command information from the upper-level intelligent agent or the user, and the device's current status information;

[0022] The vision-language backbone network is used to jointly model images and language, forming a unified semantic representation space;

[0023] The action expert module is used to generate continuous action sequences under the conditional constraints of semantic representation.

[0024] The action sequence output module is used to output device control commands within a continuous time period.

[0025] The The model specifically performs the following process:

[0026] Acquire multimodal observation data at the current moment, including visual information of the equipment operating area, current operation instructions or task descriptions, and equipment status information;

[0027] The acquired information is input into the visual-language backbone network to generate corresponding semantic representations;

[0028] Under the constraints of this semantic representation, the action expert module generates the corresponding continuous action sequence;

[0029] The generated sequence of actions is sent to the device control interface, and the device executes them sequentially to complete the corresponding physical operation task.

[0030] This invention offers the following advantages: A multimodal GUI interactive understanding and automatic operation intelligent agent system for SEM realizes end-to-end mapping from "language target / task instruction + visual observation + robot state" to "executable action sequence". The VLA model receives sample changing instructions (including sample number, target operation type, workstation / tool ​​information, etc.) from the GUI Agent and integrates sensor information such as robotic arm camera / external camera observation, joint pose, gripper state, and (optionally) force / torque, outputting continuous motion control quantities (e.g., end-effector pose increment, gripper opening and closing, tightening / loosening operation parameters). During execution, the actions are updated continuously based on real-time observations, forming an adaptive closed loop. This significantly improves the robustness and generalization ability of sample changing operations to uncertainties, overcoming the technical problems of existing fixed script / fixed trajectory control being sensitive to sample position deviation, clamping error, screw resistance fluctuation, tool wear, field of view obstruction, etc., prone to failure, and requiring frequent manual calibration and reprogramming. The VLA model can automatically complete alignment correction, contact posture adjustment, minor pose compensation and motion replanning under the unified strategy of "see-understand-action", thereby improving the success rate of sample changing and repeatability consistency, and reducing the dependence on high-precision fixtures and strict tooling positioning. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the process of executing the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of this application provided below with reference to the accompanying drawings is not intended to limit the scope of protection of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application. The present invention will be further described below with reference to the accompanying drawings.

[0033] This invention specifically relates to an intelligent agent system for multimodal GUI interaction understanding and automated operation in various SEM (Series Electron Microscopy) systems. By constructing a GUI intelligent agent based on a Visual-Language Model (VLM) and a device operation intelligent agent (robotic arm intelligent agent) based on a Visual-Language-Motion Model (VLA), it achieves collaborative control of the SEM acquisition software interface and the physical operation process of the electron microscope. The system dynamically analyzes the GUI interface state, operation commands, and task objectives through multimodal perception and semantic understanding. It also introduces a communication and state feedback mechanism between intelligent agents, enabling the device operation intelligent agent to transmit operation results and state information to the GUI intelligent agent after completing physical operations such as closing a hatch, allowing the GUI intelligent agent to continue executing subsequent software operation processes. Compared with existing technologies, this invention does not rely on predefined interfaces or fixed scripts. Instead, through multi-agent collaboration and cross-layer communication, it achieves orderly connection and adaptive control of SEM software and device layer operations, improving the automation, stability, and versatility of complex operation processes.

[0034] like Figure 1 As shown, the system specifically performs the following steps:

[0035] S1. The GUI agent receives the start signal and sends a sample change command to the robotic arm. The VLA model on the robotic arm receives the relevant signals from the GUI agent.

[0036] The S2 and VLA models output corresponding parameters based on the received instructions to control the robotic arm to open the hatch, loosen and remove the fixing screws, and take out the sample stage.

[0037] S3. The agent deployed on the computer monitor then issues instructions to the robotic arm VLA agent based on the current state to retrieve the target sample and place the sample at the designated position on the sample stage.

[0038] S4. The robotic arm reinstalls the sample stage into the microscope cavity and closes the hatch, sending a "ready" signal back to the GUI agent.

[0039] After receiving the "ready" signal, the S5 and GUI agents start the SEM software automation, performing end sample replacement and necessary initialization / status checks.

[0040] S6. The GUI agent invokes the pre-trained region selection model to select candidate observation regions (including navigation, field of view, and magnification) on the sample.

[0041] The S7 GUI agent performs focus / image quality optimization operations in the candidate area and acquires a preview image.

[0042] In S8, the GUI agent inputs the preview image into the Visual Language Model (VLM) to determine the target features; if the features are not satisfied, it returns to S6 and S7 for iterative iteration.

[0043] S9. Once the target characteristics are determined to be met, the GUI agent performs image capture, completes image saving and data export.

[0044] S10. After the data acquisition is completed, the GUI agent will reset the software and instrument status and send a success command to the robotic arm as a completion marker.

[0045] Furthermore, the Visual Language Model (VLM) in the GUI agent serves as the core of multimodal understanding and decision-making. It is used to jointly model the SEM preview image, (optional) SEM software interface screenshot, and structured parameters such as magnification / working distance / acceleration voltage with the task target text, thereby realizing a closed loop of "target feature recognition - structured output of results - driving subsequent automatic operations".

[0046] The implementation principle is as follows: First, the visual encoder converts the input image into a sequence of visual features (visual tags); then, the visual-language fusion module compresses the visual features and aligns them to the latent space of the language model; finally, the language model performs cross-modal inference on the unified tag sequence and outputs the judgment result and operation suggestions in an autoregressive manner. VLM adopts a three-module architecture of "visual encoder + fusion unit + large language model". Among them, the visual side can use the SigLIP-2 visual encoding model as the visual encoder; the fusion side uses a two-layer multilayer perceptron (MLP) to compress local visual features and map them to a feature space with the same hidden size as the large language model, so that visual tags and text tags can interact in the same attention space. At the same time, the model introduces Interleaved Multimodal Rotation Position Encoding (Interleaved-MRoPE) and DeepStack (cross-layer visual injection mechanism) to enhance the model's ability to align and infer fine-grained visual information in long context scenarios.

[0047] Furthermore, the specific implementation process of the VLM model includes:

[0048] First, the current SEM preview image and related metadata are acquired, and prompts such as "target feature description, output format constraints, and stopping conditions" are written into the input image. Then, the input image is preprocessed. Next, the image and text are input into the VLM model to obtain structured output, which includes at least whether the target was hit, confidence level, target feature text description, and next action suggestions. Finally, the output is parsed and validated, and the GUI atomic operation sequence is executed accordingly. When the target is not hit, the system updates the field of view / magnification / focus parameters based on the action suggestions and acquires the preview image again, forming an iterative closed loop. When the hit threshold and image quality threshold are met, the image is saved and returned to the initial state to complete one automatic sampling task.

[0049] In terms of model structure and computation, vision-language fusion can be represented by a two-layer MLP: Let the local 2×2 image patch features output by the visual encoder be... spliced ​​together The resulting visual token is:

[0050] ,

[0051] in For learnable projection matrices, For bias terms, It is a non-linear activation function. The dimension of this dimension is consistent with the hidden dimension of the language model, and it is used for joint reasoning with the text token in subsequent attention layers. Its physical / engineering significance lies in: By representing local image morphology information (texture, edges, grain contours, pores / cracks, and other high-frequency structures), MLP performs "local visual information compression and alignment" to ensure that image information with different resolutions and different fields of view can be input into the language model in a unified token format.

[0052] The core inference computation within the language model is based on the Transformer self-attention mechanism: for a certain layer input H, let... , indicating the first The input hidden state sequence of the Transformer layer (containing the concatenation of text tokens and visual tokens), where N is the total number of tokens and d is the hidden dimension, has ,in These represent query / key / value vectors, respectively. For learnable parameters, For each attention head, the feature dimensions, The scaling factor is used to stabilize the softmax value. The engineering significance of this formula is that the model can assign attention weights to different visual tokens based on the task text ("required target shape / size range / defect type"), thereby achieving "retrieval of image evidence based on target conditions and making decisions".

[0053] The model output is generated using autoregression: in the first... Step, the probability of the language model for the next output token is:

[0054] ,

[0055] in For multimodal input, In hidden state, To output the projection matrix, by constraining the output format to JSON in the prompt words, the generated results can be stably parsed into fields such as "hit / confidence / coordinates / action".

[0056] When the task involves feature point or bounding box localization, normalized coordinate output can be used to enhance cross-resolution robustness. The model outputs normalized coordinates. (A unified coordinate system used for grounding scenes, mapped to pixel coordinates as follows:)

[0057] ,

[0058] in The image's pixel width and height; further combined with SEM calibration ratio. (unit (can be converted to physical coordinates)

[0059] ,

[0060] in, The physical meaning of is the actual length corresponding to the pixel size, which can be obtained by SEM ruler or instrument calibration. It is used to directly convert the model positioning results into stage movement, scanning window setting or ROI (region of interest) rescanning parameters to achieve executable closed-loop control.

[0061] Furthermore, the algorithm model for the intelligent agent used in electron microscopy physics operations in this invention adopts a Vision-Language-Action (VLA) network structure, specifically built based on the π0 model. This model maps multimodal observation information and high-level operational semantics into continuously executable device control actions. This model can generate continuous, smooth, and executable action sequences based on visual observations and verbal commands without human intervention, making it suitable for precision instrument operation scenarios.

[0062] The model uses a pre-trained vision-language model as its backbone network to jointly semantically model image information and language instructions, thereby achieving a high-level understanding of the operational environment and task objectives. Building upon this, the model introduces an independent action generation sub-network to further map the semantic understanding results to a continuous action space, achieving closed-loop control from "understanding" to "execution." Unlike traditional methods that discretize or progressively predict actions, this model directly generates action sequences over a period of time through continuous action distribution modeling, improving the stability and coherence of action execution.

[0063] Furthermore, The model mainly includes the following components:

[0064] (1) A multimodal input encoding module, used to receive and encode image data from the device's vision sensor, language command information from the upper-level intelligent agent or user, and the device's current status information;

[0065] (2) Visual-language backbone network, used to jointly model images and language to form a unified semantic representation space to support the understanding and reasoning of complex operational intentions;

[0066] (3) Action expert module, which is used to generate continuous action sequences under the conditional constraints of semantic representation. This module performs parameter modeling independently of the vision-language backbone network and is specifically optimized for device control actions;

[0067] (4) Action sequence output module, used to output device control instructions within a continuous time period to support high-frequency, smooth physical operation.

[0068] Furthermore, The model's working steps include: first, acquiring multimodal observation data at the current moment, including visual information of the device's operating area, current operation instructions or task descriptions, and the device's own state information; then, inputting the above information into the vision-language backbone network to generate corresponding semantic representations; next, under the constraints of these semantic representations, generating corresponding continuous action sequences by the action expert module; and finally, sending the generated action sequences to the device control interface for the device to execute sequentially, thereby completing the corresponding physical operation tasks.

[0069] In terms of applicability, the model can acquire general operational capabilities through pre-training, or it can be fine-tuned for specific electron microscope structures or operational procedures to improve its adaptability to specific equipment and scenarios. During the inference phase, the model does not rely on fixed scripts or manual rules, but instead makes dynamic decisions based on real-time observation data, enabling it to adapt to changes in the operating environment and the needs of multi-step continuous tasks.

[0070] In the electron microscope operating agent of this invention, the action generation process employs a continuous action modeling method based on conditional flow matching, used to generate a continuous and executable sequence of device control actions under given multimodal observation conditions. This method achieves a smooth mapping from a random initial state to the target action sequence by modeling the continuous evolution of the action distribution.

[0071] Let 𝑘 represent the discrete control step:

[0072] ,

[0073] in, This represents image information acquired by the device's vision sensor. This indicates the corresponding language instructions or task description information. This represents the current state parameters of the device. The goal of the model is to generate a sequence of actions over a future period of time under the above observation conditions. ,in, This represents the device control action at a single moment, where L represents the length of the action sequence.

[0074] To achieve continuous modeling of action sequences, continuous variables are introduced. This is used to represent the intermediate states of an action as it evolves from a random initial state to the target action distribution. During this process, a noisy action representation is constructed:

[0075] ,

[0076] in, This is random noise that follows a Gaussian distribution, used to initialize the action generation process. The physical meaning of the above formula is: when When the value is close to 0, the action is mainly composed of random noise; when... As the value approaches 1, the action gradually approximates the actual executable target action sequence.

[0077] The model learns a vector field function Used to describe the state of an action under given observation conditions. The direction of evolution of the change. This function is output by the action expert module, and its training objective is to minimize the difference between the predicted vector field and the ideal evolution direction. Its loss function is expressed as:

[0078] ,

[0079] in, Indicates model parameters, This represents the ideal evolution direction from the current noise state to the target action sequence. The physical meaning of this loss function is to constrain the model to learn the correct action evolution path, so that the generation process can stably converge to a reasonable control action.

[0080] During the inference phase, the action generation process starts from randomly initialized action states. Initially, we proceed step by step by performing numerical integration on the aforementioned vector field, specifically in the following form:

[0081] ,

[0082] in, This is the integration step size. Through multiple iterations and updates, it eventually reaches... Obtain the complete action sequence This process can be understood as continuously "guiding" the action space under the constraints of observation conditions, thereby generating smooth, coherent control commands that conform to the motion constraints of the equipment.

[0083] The above-described features enable the present invention to perform high-precision continuous control of SEM-related equipment (such as robotic arms, door mechanisms, etc.) without the need for discretized motion space, significantly improving operational stability and repeatability, and making it suitable for multi-step, highly constrained automated operation scenarios of electron microscopes.

[0084] This invention adopts a method based on The VLA (Vision-Language-Action) model, serving as the core for motion generation and closed-loop control on the robotic arm side, achieves end-to-end mapping from "language target / task instructions + visual observation + robot state" to "executable action sequences." The VLA model receives sample-changing instructions from the GUI Agent (including sample number, target operation type, workstation / tool ​​information, etc.) and integrates robotic arm / external camera observations, joint poses, gripper states, and (optionally) force / torque sensor information. It outputs continuous motion control quantities (e.g., end-effector pose increments, gripper opening / closing, tightening / loosening operation parameters), continuously updating the actions based on real-time observations during execution to form an adaptive closed loop. This significantly improves the robustness and generalization ability of sample-changing operations to uncertainties, overcoming the technical problems of existing fixed-script / fixed-trajectory control being sensitive to sample position deviations, clamping errors, screw resistance fluctuations, tool wear, and field-of-view obstruction, prone to failure, and requiring frequent manual calibration and reprogramming. The VLA model, under a unified "see-understand-act" strategy, automatically completes alignment correction, contact posture adjustment, minor pose compensation, and motion replanning, thereby improving sample change success rate and repeatability consistency, and reducing reliance on high-precision fixtures and strict tooling positioning. The anomaly self-recovery and safety enhancements brought by the VLA model overcome the problems of "difficult anomaly detection, difficult reversal, and easy collision / jamming" in existing unattended sample change processes. By breaking down the execution process into verifiable sub-skills (opening the hatch, unscrewing screws, picking up and placing the sample stage, picking up and placing the sample, resetting and closing the hatch, etc.) and introducing status confirmation conditions at each step (e.g., successful gripping, screw torque reaching the threshold, sample stage in place, hatch closure completed, etc.), the system can automatically take reversal actions, realign, or enter a safety shutdown procedure when it detects deviation from the target or signs of failure, reducing the risk of misoperation and protecting the instrument and sample. In collaboration with the GUI agent's software-automated closed-loop system, a seamless link is formed, encompassing "hardware resizing - software initialization - region search - focus determination - data acquisition and storage." This enables the entire system to operate stably and unattended for longer periods, improving effective acquisition time and output per unit time. Furthermore, by recording VLA output actions, key states, and results (reasons for success / failure), traceability and verifiability are achieved. Overall, this fills the gaps in existing technologies regarding general agent capabilities, closed-loop adaptation, and anomaly handling.

[0085] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and improvements, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. An intelligent agent system for multimodal GUI interactive understanding and automatic operation in SEM, characterized in that: The system includes a GUI intelligent agent set on the software side and a robotic arm intelligent agent set on the device side; The GUI agent is configured to receive a start signal, send a sample change command to the robotic arm agent, and start the SEM software automation after receiving the robotic arm agent's ready signal. At the same time, it calls the pre-trained region selection model, selects candidate observation regions on the sample, performs focusing and image quality optimization operations in the candidate regions and acquires preview images, inputs the preview images into the VLM model for target feature determination, takes a picture and saves the image when the target features are met, and sends a success command to the robotic arm agent as a completion mark. The robotic arm agent is configured to receive signals sent by the GUI agent through the VLA model on the robotic arm, and output corresponding parameters to control the robotic arm to complete the corresponding operation before sending a ready signal back to the GUI agent. The GUI agent uses the VLM model to jointly model the SEM preview image, SEM software interface screenshot, and structured parameters with the task target text, thereby achieving a closed loop of target feature recognition, structured result output, and driving subsequent automatic operations. The GUI agent uses a VLM model to jointly model SEM preview images, SEM software interface screenshots, structured parameters, and task target text, including: Collect the current SEM preview image and related metadata, and write the set words into the prompt words; The input image is preprocessed, and then the image and text are input into the VLM model to obtain a structured output that includes at least whether the target is hit, confidence score, target feature text description, and next action suggestion; The output is parsed and verified, and the GUI atomic operation sequence is executed accordingly. When the system determines that the target has not been hit, it updates the field of view, magnification, and focus parameters based on the action suggestions and re-captures the preview image, forming an iterative closed loop. When the hit threshold and image quality threshold are met, the image saving process begins, and the system returns to the initialization state to complete an automatic sampling task.

2. The intelligent agent system for multimodal GUI interactive understanding and automatic operation in SEM according to claim 1, characterized in that: The process of inputting images and text into the VLM model to obtain a structured output that includes at least whether the target was hit, confidence level, target feature text description, and next action suggestion includes: Image and text are input together into the VLM model. Visual-language fusion is expressed through a two-layer multilayer perceptron (MLP). Let the local 2×2 image patch features output by the visual encoder of the VLM model be... spliced ​​together The fused visual marker is ,in, For learnable projection matrices, For bias terms, It is a non-linear activation function. Represents local topographic information of the image, i=1, 2, 3, 4; For input data H containing text and visual markers at a certain layer, the formula is used... By assigning attention weights to different visual markers based on the task text, image evidence can be retrieved according to target conditions, and a decision can be made. These represent the query vector, key vector, and value vector, respectively. For learnable parameters, For each attention head, the feature dimensions, This is the scaling factor; The VLM model output is generated through autoregression. At step t, the language model's probability for the next output visual token is... By constraining the output format in the prompts, the generated results are stably parsed into a structured output that includes whether the target was hit, confidence level, target feature text description, and next action suggestion. For multimodal input, In hidden state, This is for outputting the projection matrix.

3. The intelligent agent system for multimodal GUI interactive understanding and automatic operation in SEM according to claim 1, characterized in that: The VLA model is based on Model building is used to map multimodal observation information and high-level operational semantics into continuously executable device control actions; The model includes a multimodal input encoding module, a visual-language backbone network, an action expert module, and an action sequence output module; The multimodal input encoding module is used to receive and encode image data from the device's vision sensor, language command information from the upper-level intelligent agent or the user, and the device's current status information; The vision-language backbone network is used to jointly model images and language, forming a unified semantic representation space; The action expert module is used to generate continuous action sequences under the conditional constraints of semantic representation. The action sequence output module is used to output device control commands within a continuous time period.

4. The intelligent agent system for multimodal GUI interactive understanding and automatic operation in SEM according to claim 3, characterized in that: The The model specifically performs the following process: Acquire multimodal observation data at the current moment, including visual information of the equipment operating area, current operation instructions or task descriptions, and equipment status information; The acquired information is input into the visual-language backbone network to generate corresponding semantic representations; Under the constraints of this semantic representation, the action expert module generates the corresponding continuous action sequence; The generated sequence of actions is sent to the device control interface, and the device executes them sequentially to complete the corresponding physical operation task.

Images