A natural language driven SEM scanning task processing system

Abstract

The embodiment of the application relates to a natural language driven SEM scanning task processing system, the system comprising: a parameter identification intelligent agent, a grid planning engine, a task planning intelligent agent, an image splicing engine, an analysis intelligent agent, an interactive terminal, a scheduling service, a cache service, a driving service and SEM equipment; the interactive terminal is connected with the parameter identification intelligent agent, the grid planning engine, the task planning intelligent agent, the image splicing engine, the analysis intelligent agent, the scheduling service and the cache service respectively; the driving service is connected with the scheduling service, the cache service and each SEM equipment respectively; and the scheduling service is further connected with the cache service. Based on the application, the task processing quality and the task processing efficiency can be improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a natural language-driven SEM scanning task processing system. Background Technology

[0002] Scanning Electron Microscopy (SEM) is a core tool for characterizing micro- and nano-scale structures. Its basic principle is to use a high-energy electron beam focused by a magnetic lens to scan point-by-point on the sample surface or interior, generating corresponding electron micrographs based on the scanning results. Current SEM scanning processes have several problems: 1. Scanning parameters and workflows can only be manually set and cannot be automatically generated. 2. Interfaces of devices from different manufacturers are incompatible, requiring significant driver integration work for upper-level application development to ensure compatibility with all manufacturers' devices. 3. In scenarios requiring cross-scale analysis, such as nanomaterials and semiconductor devices, a single high-magnification field of view is insufficient to cover macroscopic morphology, typically employing a "grid scanning + post-stitching" method. However, traditional stitching methods are prone to problems such as image misalignment, stitching gaps, and error propagation when dealing with the inherent nonlinear distortions of electron microscopy, making it difficult to obtain high-fidelity complete images. 4. Feature extraction and recognition after image acquisition are often offline, lacking closed-loop feedback, resulting in low task processing efficiency.

[0003] To address the aforementioned issues, we propose an improved solution: multiple intelligent processing modules (parameter recognition agent, grid planning engine, task planning agent, image stitching engine, and analysis agent) are introduced into the traditional SEM processing system. A unified, standardized interface set is provided to upper-layer applications through a driver service. Specifically, the parameter recognition agent automatically generates scanning parameters based on multimodal image and text information; the grid planning engine automatically plans the scanning grid; the task planning agent provides an automatic task workflow generation mechanism driven by natural language; the image stitching engine addresses image distortion; the unified interface driver service simplifies the development difficulty and workload of upper-layer applications; and the analysis agent, centered on the Uni-AIMS analysis tool, performs online real-time analysis of electron microscope images. The technical problem this invention aims to solve is how to implement this improved solution. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by providing a natural language-driven SEM scanning task processing system. This system includes: a parameter recognition agent, a grid planning engine, a task planning agent, an image stitching engine, an analysis agent, an interactive terminal, a scheduling service, a caching service, a driving service, and an SEM device. The parameter recognition agent is used for multimodal feature extraction, joint feature encoding, and scanning parameter prediction based on image and text information. The grid planning engine is used to optimize the overlap rate based on the ROI region map, ROI region parameters, single-frame field of view size, and overlap rate, and to plan the scanning grid based on the optimized overlap rate. The task planning agent uses a thought chain reasoning mechanism to perform multi-step deduction based on natural language text and its corresponding historical context. During the deduction process, it performs atomic operation sequence parsing of the electron microscope equipment operation steps for the current scanning task, generates a task step flow in a specified format based on the parsed sequence, and performs compliance verification on the task step flow. The image stitching engine is used to perform sub-image correction and full-sequence sub-image stitching based on target region parameters, a scanning grid table, a high-magnification sub-image sequence, and a low-magnification panoramic image to obtain the corresponding high-fidelity panoramic image. The analysis agent uses the Uni-AIMS analysis tool to perform scale recognition, particle and pore target segmentation, particle size recognition, pore size recognition, particle distribution analysis, and pore distribution analysis on the electron microscope images to be analyzed. The driver service provides a unified set of standardized interfaces and adapts each standardized interface to the corresponding manufacturer's interface for each SEM device. It also facilitates the intermediate forwarding from upper-layer applications (such as scheduling services and caching services) to lower-layer devices (such as SEM devices) through the standardized interface set. The interaction interface generates a corresponding interactive page (components: dialog input box, history dialog area, status panel, electron microscope window) for the current scan task when the user confirms the creation of a new scan task. Based on the user's submission information and parameters on the current page, it automatically completes parameter settings and task step flow generation using the identification agent, grid planning engine, and task planning agent. Through interaction with the scheduling service, caching service, driver service, and the SEM device's backend, it automatically completes the scan task. At the end of the task, it automatically calls the image stitching engine to stitch the scanned sub-images into a high-fidelity panoramic image and uses the analysis agent to perform online real-time analysis of the high-fidelity panoramic image. Based on this invention, the problem of manual dependence on parameter setting and task flow customization can be solved, the development difficulty and workload of upper-layer applications can be simplified, the distortion phenomenon of large field-of-view mosaic images can be improved, and online real-time analysis of electron micrographs can be performed, thereby reducing the difficulty of task processing, improving the quality of task processing, optimizing the effect of task processing, and improving the efficiency of task processing.

[0005] To achieve the above objectives, embodiments of the present invention provide a natural language-driven SEM scanning task processing system, the system comprising: a parameter recognition agent, a grid planning engine, a task planning agent, an image stitching engine, an analysis agent, an interactive terminal, a scheduling service, a caching service, a driving service, and an SEM device; The interactive terminal is connected to the parameter recognition agent, the grid planning engine, the task planning agent, the image stitching engine, the analysis agent, the scheduling service, and the caching service, respectively; the driving service is connected to the scheduling service, the caching service, and each of the SEM devices, respectively; the scheduling service is also connected to the caching service. The parameter recognition agent is used to receive parameter recognition requests; and the preset parameter recognition model performs multimodal feature extraction, joint feature encoding, and scanning parameter prediction based on the scan overview text and reference electron microscope image carried in the current request to obtain the corresponding predicted parameter set; and substitutes the predicted parameter set into the preset configuration template to generate the corresponding scan parameter configuration and sends it back to the current requester; The grid planning engine receives grid planning requests and identifies whether the ROI region map carried in the current request is empty. If it is, the overlap rate carried in the current request is used as the corresponding optimized overlap rate. If not, the preset overlap rate optimization model optimizes the overlap rate based on the ROI region map, ROI region parameters, single-frame field of view size, and overlap rate carried in the current request to obtain the corresponding optimized overlap rate. The engine then performs scanning grid planning based on the optimized overlap rate to obtain the corresponding scanning grid table and sends the optimized overlap rate and the scanning grid table back to the current requester. The task planning agent receives task planning requests and inputs the planning instruction text carried in the current request and its corresponding historical context into a preset prompt instruction template to generate corresponding task planning instructions. The task planning instructions are then input into a preset scanning task planning model. The model, based on the instruction prompts and a thought chain reasoning mechanism, uses the historical context as a reference to perform multi-step deduction based on the planning instruction text. During the deduction process, atomic operation sequence parsing is performed on the electron microscope equipment operation steps of the current scanning task, and a task step flow in a specified format is generated based on the parsed sequence. The task step flow undergoes compliance verification, and the verified task step flow is sent back to the current requester. The image stitching engine is used to receive image stitching requests; and to perform sub-image correction and full-sequence sub-image stitching based on the target area parameters carried in the current request, the scanning grid table, the high magnification sub-image sequence, and the low magnification panoramic image to obtain the corresponding high-fidelity panoramic image and send it back to the current requester; The analytical agent is used to receive graph analysis requests; and input the electron microscope image to be analyzed carried in the current request into a preset Uni-AIMS analysis tool for processing to obtain the corresponding electron microscope image analysis results, which are then sent back to the current requester. The interactive terminal is used to generate a corresponding task interaction page for the current scanning task when the user confirms the creation of a new scanning task for a certain SEM device through the client menu. The components of the task interaction page include at least a dialog input box, a history dialog area, a status panel, and an electron microscope window. Based on the submission information entered by the user on the current task interaction page, and the parameter recognition agent, the grid planning engine, and the task planning agent, a step flow generation process is performed to obtain the corresponding task step flow. The device identifier of the current SEM device and the task step flow are sent to the scheduling service. Before receiving the end notification from the scheduling service, the status panel is updated based on the global status information pushed by the caching service. After receiving the end notification, an image stitching request is generated based on the cached image from the caching service and sent to the image stitching engine. The high-fidelity panoramic image returned by the image stitching engine is used as the electron microscope image to be analyzed. The image analysis request carrying the electron microscope image to be analyzed is sent to the analysis agent. The current task interaction page is updated based on the electron microscope image analysis results returned by the analysis agent. Upon receiving the device identifier and the task step flow, the scheduling service creates a set of corresponding execution instruction queues, execution feedback queues, status buffers, and image buffers for the current scanning task on the caching service; and calls the SEM device corresponding to the device identifier through the driving service to execute the current task step flow step by step, and sends the end notification to the interactive terminal when the current task ends. The caching service is used to store the corresponding execution instruction queue, execution feedback queue, status cache area, and image cache area for all scanning tasks; and to push the global status information of each status cache area to the corresponding interactive terminal at a preset push frequency. The driving service provides a unified set of standardized interfaces and adapts each standardized interface to the corresponding manufacturer's equipment interface of each SEM device. It also forwards the instruction information sent by the scheduling service to the designated SEM device through the standardized interface set and writes the instruction execution feedback, global status information or electron microscope image sent by each SEM device into the corresponding execution feedback queue, status cache or image cache through the standardized interface set. The SEM device is used to perform single or multiple instruction operations based on the received instruction information to obtain corresponding instruction execution feedback, and write the instruction execution feedback into the corresponding execution feedback queue through the driving service; and update the global status information of the current scanning task according to a preset status update frequency, and write the latest global status information into the corresponding status cache through the driving service; and after each grid cell scan is completed and the corresponding electron microscope image is generated, write the latest electron microscope image into the corresponding image cache through the driving service.

[0006] Preferably, the prediction parameter set includes at least the target region parameters, the single-frame field of view size, the magnification range, the overlap rate, the acceleration voltage, the focus type, and the detector type; the target region parameters consist of the coordinates of the four vertices of the target region; the single-frame field of view size consists of the corresponding field of view width and field of view height; the focus type includes autofocus and manual focus; the detector type includes SE type, BSE type, and EDS type. The scan grid table consists of multiple scan grid records; each scan grid record consists of the single-frame field of view size and the coordinates of the grid center point. The task step flow is composed of multiple single-step operation nodes arranged sequentially; each single-step operation node includes a node identifier, a node type, and a node instruction set; the node type includes single-instruction nodes and multi-instruction nodes; the node instruction set consists of one or more first instructions; the first instruction includes an instruction code and instruction parameters; when the node type is a single-instruction node, the total number of instructions in the node instruction set is 1; when the node type is a multi-instruction node, the total number of instructions in the node instruction set is greater than 1. The scanning task planning model is based on a large language model or a multimodal large model, including at least the GPT series model, the DeepSeek series model, the SenseNova-MARS series model, the Wenxin series model, and the Tongyi Qianwen Qwen series model. The key paragraphs of the prompt instruction template include at least five parts: model role description paragraph, historical dialogue paragraph, user requirement paragraph, inference chain requirement paragraph, and output format constraint paragraph; wherein, the model role description paragraph, the inference chain requirement paragraph, the output format constraint paragraph, and the step instruction constraint paragraph are each a fixed natural language text; the historical dialogue paragraph and the user requirement paragraph are configurable paragraphs of the template, the historical dialogue paragraph is configured based on the historical context input to the template, and the user requirement paragraph is configured based on the planning instruction text input to the template; The model role description text is used to set the task role of the scanning task planning model as an SEM intelligent control assistant; and to describe the role task corresponding to the model: using the historical dialogue text as a context reference, and according to the thought chain reasoning steps set by the reasoning chain requirement text, multi-step deduction is performed according to the user requirement text, and atomic operation sequence parsing is performed on the electron microscope equipment operation steps during the deduction process, and the task step flow in the format specified by the output format constraint text is generated based on the parsing sequence. The reasoning chain requirement section is used to describe step-by-step reasoning steps of the thought chain of the scanning task planning model; the output format constraint section is used to describe the formatted text format of the task step flow output by the scanning task planning model, and the formatted text format includes at least TypeScript format and JSON format. The high-magnification sub-image sequence consists of multiple high-magnification sub-images; each high-magnification sub-image in the high-magnification sub-image sequence corresponds one-to-one with the scanning grid record in the scanning grid table, and each high-magnification sub-image is a grid electron microscope scan of a corresponding scanning grid in the target region corresponding to the target region parameter; when the low-magnification panoramic image is not empty, it is an electron microscope scan of the entire target region corresponding to the target region parameter; the scanning magnification of the high-magnification sub-images is greater than that of the low-magnification panoramic image; The Uni-AIMS analysis tool is used to perform scale identification, particle and pore target segmentation, particle size identification, pore diameter identification, particle distribution analysis, and pore distribution analysis on the electron microscope image to be analyzed. The analysis results of the electron microscope image output by the Uni-AIMS analysis tool include at least scale identification information, target segmentation mask image, statistical table of particle size / pore diameter data, and analysis chart of particle / pore distribution. The task interaction page adopts the layout style of an integrated development environment; the history dialogue area is used to display the human-computer interaction text content of the current scanning task line by line; the status panel is used to display the global status information of the current scanning task; the global status information includes at least scanning progress, vacuum status, electron beam parameters, detector status, and abnormal alarm information; the electron microscope window is used to load electron microscope images, and supports loading by selecting components or manually dragging and dropping, and supports selecting ROI areas on the electron microscope images loaded in the window by manually drawing. The execution instruction queue is used to sequentially store all the single-step operation nodes of the task step flow, with each queue message corresponding to one single-step operation node; the execution instruction queue has a configurable message read pointer for locating the current execution node; The execution feedback queue is used to store the instruction execution feedback corresponding to each single-step operation node; the instruction execution feedback includes the node identifier, the node type, and the instruction feedback set; the instruction feedback set consists of one or more first instruction feedbacks; the first instruction feedback includes the first instruction and the first feedback; when the node type is a single instruction node, the total number of instruction feedbacks in the instruction feedback set is 1; when the node type is a multi-instruction node, the total number of instruction feedbacks in the instruction feedback set is greater than 1. The state cache is used to store the global state information; The image buffer is used to store electron microscope images.

[0007] Preferably, the parameter recognition model includes a text encoder, an image encoding module, an image encoder, a target detection model, a multimodal feature joint encoding module, and a parameter prediction module; The first model input terminal of the parameter recognition model is used to receive the scan overview text, the second model input terminal is used to receive the reference electron microscope image, and the model output terminal is used to output the predicted parameter set. The text encoder's input is connected to the first model's input, and its output is connected to the first input of the multimodal feature joint encoding module; the image encoding module's input is connected to the second model's input, and its output is connected to the second input of the multimodal feature joint encoding module; the image encoding module is also connected to both the image encoder and the target detection model; the multimodal feature joint encoding module's output is connected to the parameter prediction module's input; and the parameter prediction module's output is connected to the model's output. The text encoder is used to extract text features from the scanned overview text to obtain the corresponding feature vector X1, which is then sent to the multimodal feature joint encoding module. The feature vector X1 has a shape of 1×D1, where D1 is a preset first feature dimension. The image encoding module is used to identify whether the reference electron microscope image is empty. If the reference electron microscope image is empty, the corresponding feature vectors X2 and X3 are set to all zero vectors. If the reference electron microscope image is not empty, the image encoder is called to perform global feature extraction on the reference electron microscope image to obtain the corresponding feature vector X2. Using the ROI region bounding box as the detection target, the target detection model is called to perform target detection on the reference electron microscope image to obtain a set of corresponding target detection boxes. The target detection box with the highest confidence in the target detection box set is taken as the first matching box, and the first matching box is... The system identifies whether the bounding box is empty. If it is, the corresponding feature vector X3 is set to an all-zero vector; otherwise, the feature vector X3 is composed of the center point coordinates, width, and height of the first matching bounding box. The obtained feature vectors X2 and X3 are then sent to the multimodal feature joint encoding module. The shape of feature vector X2 is 1×D2, and the shape of feature vector X3 is 1×D3, where D2 and D3 are the preset second and third feature dimensions, respectively, and D3=4. The image encoder is implemented based on a residual network model, and the target detection model is implemented based on the YOLO series models. The multimodal feature joint encoding module is used to map the feature vectors X1, X2, and X3 to a preset target feature space using three linear layers, denoted as feature vectors Y1, Y2, and Y3 respectively; and to concatenate the feature vectors Y1, Y2, and Y3 to obtain the corresponding feature vector Y4; and to use an MLP model to encode the feature vector Y4 to obtain the corresponding feature vector Y5, which is then sent to the parameter prediction module. The mapping methods for feature vectors Y1, Y2, and Y3, and the encoding method for feature vector Y5 are as follows: , , ; ; W1 and b1 are the weight matrix and bias vector of the first linear layer. The weight matrix W1 has a shape of D4×D1, and the bias vector b1 has a shape of 1×D4, where D4 is the feature dimension of the target feature space. W2 and b2 are the weight matrix and bias vector of the second linear layer. The weight matrix W2 has a shape of D4×D2, and the bias vector b2 has a shape of 1×D4. W3 and b3 are the weight matrix and bias vector of the third linear layer. The weight matrix W3 has a shape of D4×D3, and the bias vector b3 has a shape of 1×D3. The feature vectors Y1, Y2, and Y3 all have a shape of 1×D4. The feature vector Y4 has a shape of 1×3D4. MLP1() is the inference function of the first MLP model. The parameter prediction module is used to identify whether the first matching box is empty. If it is, the corresponding target region parameter is set to the default region parameter of the SEM device corresponding to the current scanning task. Otherwise, the coordinates of the four vertices are calculated based on the center point coordinates, width, and height of the first matching box, and the four coordinates are used to form the corresponding target region parameter. The corresponding linear regression prediction head is used to predict the single-frame field of view size, the magnification range, the overlap rate, and the acceleration voltage. The corresponding classification prediction head is used to predict the focus type and the detector type. The corresponding prediction parameter set is formed by the obtained target region parameter, single-frame field of view size, magnification range, overlap rate, acceleration voltage, focus type, and detector type and then output.

[0008] Preferably, the overlap rate optimization model includes a first feature extraction network, a second feature extraction network, a feature fusion layer, a feature encoding layer, an overlap rate prediction layer, and a prediction constraint module; The first model input terminal of the overlap rate optimization model is used to receive the ROI region map, the second model input terminal is used to receive the ROI region parameters, the single frame field of view size and the overlap rate, and the model output terminal is used to output the optimized overlap rate. The input of the first feature extraction network is connected to the input of the first model, and its output is connected to the first input of the feature fusion layer; the input of the second feature extraction network is connected to the input of the second model, and its output is connected to the second input of the feature fusion layer; the output of the feature fusion layer is connected to the input of the feature encoding layer; the output of the feature encoding layer is connected to the input of the overlap rate prediction layer; the output of the overlap rate prediction layer is connected to the input of the prediction constraint module; and the output of the prediction constraint module is connected to the model output. The first feature extraction network is implemented based on a residual network model; the first feature extraction network is used to perform global feature extraction on the ROI region map to generate the corresponding feature vector X. 1 Send to the feature fusion layer; The second feature extraction network is implemented based on an MLP model; the second feature extraction network is used to form an initial vector composed of the ROI region parameters, the single-frame field of view size, and the overlap rate; and to extract features from the initial vector to generate a corresponding feature vector X. 2 Send to the feature fusion layer; The feature fusion layer is used to perform vector concatenation on the feature vector X. 1 and the feature vector X 2Vector concatenation yields the corresponding feature vector X. 3 Send to the feature coding layer; The feature encoding layer is implemented based on an MLP model; the feature encoding layer is used to process the feature vector X. 3 Feature encoding is performed to obtain the corresponding feature vector X. 4 Send to the overlap rate prediction layer; The overlap rate prediction layer is implemented based on a linear regression prediction network; the overlap rate prediction layer is used to predict the overlap rate based on the feature vector X. 4 The corresponding predicted overlap rate ρ is obtained by performing overlap rate prediction. * Send to the prediction constraint module; The prediction constraint module records the overlap rate of the model input as ρ. old And based on the overlap rate ρ old And a preset percentage α relative to the predicted overlap rate ρ * The corresponding optimized overlap rate ρ is obtained by applying constraints. new : ; Where 0 < α < 1.

[0009] Preferably, the grid planning engine is specifically used when obtaining the corresponding scan grid table by performing scan grid planning based on the optimized overlap rate: The center point coordinates of each scan grid on each grid row of the ROI region are calculated based on the ROI region parameters, the single-frame field of view size, and the optimized overlap rate to obtain the corresponding grid center point coordinates; and a corresponding scan grid record is formed by each grid center point coordinate and the single-frame field of view size; and all the scan grid records are sorted according to the scanning order to form the corresponding scan grid table.

[0010] Preferably, the task planning agent is specifically used when performing compliance checks on the task step flow: A sequential traversal is performed on all single-step operation nodes of the task step flow; during this traversal, the currently traversed single-step operation node is designated as the current node; and a poll is performed on each of the first instructions of the current node; during this poll, the currently polled first instruction is designated as the current instruction; and the compliance of the instruction code and instruction parameters of the current instruction is verified based on a preset instruction rule set; if the current instruction verification is successful, the process moves to the next first instruction and continues polling until all the first instructions of the current node have been polled; if the current instruction verification fails, the current polling and traversal ends, and the corresponding step flow verification result is set to failure; and After all the polling operations in this round are successfully verified, the process moves to the next single-step operation node and continues traversing until all single-step operation nodes have been traversed. When all instructions in all single-step operation nodes are successfully verified, the corresponding step flow verification result is set to success. The instruction rule set includes multiple instruction rules. Each instruction rule corresponds one-to-one with an instruction code. Each instruction rule is used to define the instruction code encoding of its corresponding instruction, the name, data type, and parameter value range constraints of each instruction parameter of its corresponding instruction, whether its corresponding instruction has preconditions, and, if its corresponding instruction has preconditions, the logical judgment rules corresponding to the preconditions. The system then identifies whether the obtained step flow verification result is successful; if yes, the compliance check is confirmed to have passed; otherwise, the compliance check is confirmed to have failed.

[0011] Preferably, the image stitching engine is specifically used when the corresponding high-fidelity panoramic image is obtained by performing sub-image correction and full-sequence sub-image stitching based on the target area parameters carried in the current request, the scan grid table, the high-magnification sub-image sequence, and the low-magnification panoramic image, and then sending it back to the current requester: Step 71: When the low-magnification panoramic image is not empty, based on the target region parameters and the scanning grid table, the corresponding low-magnification sub-images of each high-magnification sub-image sequence in the low-magnification panoramic image are confirmed, and distortion correction is performed on the corresponding high-magnification sub-images based on the feature point matching relationship, specifically: Each high-magnification sub-image in the high-magnification sub-image sequence is taken as the current high-magnification sub-image; the magnification corresponding to the current high-magnification sub-image is taken as the current high-magnification; the magnification corresponding to the low-magnification panoramic image is taken as the current low-magnification; the scan grid record corresponding to the current sub-image in the scan grid table is taken as the current record, and the scan grid region corresponding to the current record in the low-magnification panoramic image is confirmed based on the target region parameters, the single-frame field of view size of the current record, and the grid center point coordinates, and the sub-image of the current scan grid region is extracted as the current low-magnification sub-image; The system performs feature point detection on the current high-magnification sub-image and the current low-magnification sub-image respectively according to a preset feature point detection algorithm to obtain the corresponding high-magnification sub-image feature point set and low-magnification sub-image feature point set; and confirms all matching feature point pairs of the high-magnification sub-image feature point set and the low-magnification sub-image feature point set based on a preset feature point matching algorithm; and identifies the projection transformation relationship between the current high-magnification sub-image and the current low-magnification sub-image based on all the obtained matching feature point pairs; and performs pixel coordinate transformation on the current high-magnification sub-image based on the identified projection transformation relationship; wherein, the feature point detection algorithm includes at least the SIFT algorithm, SURF algorithm, and ORB algorithm; and the feature point matching algorithm includes at least the Brute-Force algorithm and FLANN algorithm; Step 72: Construct a scanning grid plane space, and identify all adjacent sub-image pairs of the high-magnification sub-image sequence in the scanning grid plane space to obtain the corresponding set of adjacent sub-image pairs, specifically: The scanning grid plane space is constructed based on the target region parameters, the known single-frame field of view size, and the overlap rate; and all spatially adjacent sub-images of each high-magnification sub-image in the high-magnification sub-image sequence are retrieved based on the scanning grid plane space; and each high-magnification sub-image and its corresponding adjacent sub-image form a corresponding adjacent sub-image pair; all the obtained adjacent sub-image pairs are deduplicated; and all the deduplicated adjacent sub-image pairs form a corresponding adjacent sub-image pair set; The scanning grid plane space includes multiple scanning grids, the height and width of each scanning grid satisfying the single-frame field of view size; the overlap ratio of any two adjacent scanning grids in the plane space satisfies the overlap rate; each scanning grid corresponds to a high magnification sub-image; all spatially adjacent sub-images include the left neighbor sub-image, right neighbor sub-image, upper neighbor sub-image, and lower neighbor sub-image of the current sub-image; each of the adjacent sub-image pairs in the set of adjacent sub-image pairs is denoted as the corresponding sub-image pair Z. i 1 ≤ index i ≤ N, where N is the total number of subgraph pairs in the current subgraph pair set; each of the subgraph pairs Zi The two high-magnification sub-graphs are denoted as the corresponding sub-graph P. i,1 P i,2 ; Step 73, calculate the Z values ​​for each of the subgraph pairs based on the phase correlation method. i The corresponding relative displacement s i And by obtaining all the said relative displacements s i The corresponding set of relative displacements is as follows: Each of the subgraphs is paired with Z. i As the current subgraph pair; and for the subgraph P of the current subgraph pair i,1 P i,2 Performing a Fourier transform yields the corresponding spectrum F. i,1 Spectrum F i,2 ; and based on the spectrum F i,1 and the spectrum F i,2 F i,2 Calculate the corresponding cross-power spectrum. , For the spectrum F i,2 The complex conjugate of ε is a preset small constant to prevent the denominator from being zero; and the inverse Fourier transform of the cross-power spectrum R is performed to obtain the corresponding correlation function r(x,y), where x and y are the sub-graphs P. i,1 P i,2 The lateral and longitudinal relative displacements; and the extreme points (x, y) of the related function r(x, y). peak ,y peak Solve the problem; and calculate the lateral relative displacement x of the extreme points obtained from the solution. peak Longitudinal relative displacement y peak Let x be the corresponding lateral relative displacement. i Longitudinal relative displacement y i ; and by the lateral relative displacement x i and the longitudinal relative displacement y i The corresponding relative displacements s i ; and from all the said relative displacements s obtained i Form the corresponding set of relative displacements; Step 74, pair each of the subgraphs with Z i Subgraph P i,1 P i,2 The corresponding coordinates of the grid center point are marked as the corresponding subgraph center point c. i,1 c i,2 ; and based on all the said subgraphs, Z i The corresponding subgraph center point c i,1 c i,2 and the relative displacement s iConstruct a translation optimization objective function E1; and based on a preset first optimization algorithm, determine the subgraph center point offset Δc that minimizes the translation optimization objective function E1. i,1 , △c i,2 An estimation is performed, and the coordinates of the grid center points of each of the high magnification sub-images are reset based on the estimation results; The translation optimization objective function E1 is: ; The first optimization algorithm includes the least squares method and the simulated annealing algorithm; During the optimization process, the sub-image center point offset Δc corresponding to the same high magnification sub-image is... i,1 or △c i,2 Maintaining global uniqueness; after optimization, because the center point offset △c of the sub-image corresponding to the same high magnification sub-image i,1 or △c i,2 To maintain global uniqueness, the offset of the center point of each globally unique sub-image corresponding to each high magnification sub-image is recorded as △c. Thus, the coordinates of the center point of the grid after reset for each high magnification sub-image are equal to the coordinates of the center point of the grid before reset plus △c. Step 75, perform the feature point detection algorithm on each of the sub-graph pairs Z. i Feature point detection is performed on the two subgraphs to obtain the corresponding first point set and second point set; and based on the feature point matching algorithm, all matching point pairs in the first point set and the second point set are confirmed to obtain the corresponding matching point pair set D. i ; Wherein, the set of matching point pairs D i With the subgraph pair Z i One-to-one correspondence; the set of matching point pairs D i Includes multiple matching point pairs d i,j 1 ≤ index j ≤ M, where M is the total number of matching point pairs in the current matching point pair set; each of the matching point pairs d i,j From the coordinates of point p i,j,1 Point coordinates p i,j,2 Composition; the point coordinates p i,j,1 For the corresponding subgraph P i,1 The coordinates of the point, the coordinates of the point p i,j,2 For the corresponding subgraph P i,2 The coordinates of the point; Step 76: Initialize a corresponding linear transformation function for each of the high magnification sub-images in the high magnification sub-image sequence as the initialization projection transformation relationship T. u 1 ≤ index u ≤ K, where K is the total number of subgraphs in the current subgraph sequence; and based on all the aforementioned projection transformation relationships Tu and the set D of all the said matching point pairs i Construct a nonlinear distortion optimization objective function E2; and based on a preset second optimization algorithm, optimize all projection transformation relationships T that minimize the nonlinear distortion optimization objective function E2. u Make an estimate; Wherein, each of the projection transformation relationships T u Used to perform coordinate projection of the scanning grid plane space onto the corresponding high magnification sub-image; The nonlinear distortion optimization objective function E2 is: ; Let p be the coordinates of the point. i,j,1 The corresponding subgraph P i,1 The corresponding projection transformation relationship T u , Let p be the coordinates of the point. i,j,1 Through projection transformation relationship The coordinates of the projection point obtained after calculation; Let p be the coordinates of the point. i,j,2 The corresponding subgraph P i,2 The corresponding projection transformation relationship T u , Let p be the coordinates of the point. i,j,2 Through projection transformation relationship The coordinates of the projection point obtained after calculation; The second optimization algorithm includes the Levenberg-Marquardt algorithm, the Gauss-Newton algorithm, and the simulated annealing algorithm; Step 77, based on each estimated projection transformation relationship T u The pixel coordinates of the corresponding high magnification sub-image are transformed by the pixel coordinates in the scanning grid plane space to obtain the corresponding grid plane sub-image; and the obtained grid plane sub-images are stitched together in spatial arrangement to obtain the corresponding high-fidelity panoramic image, which is then sent back to the current requester.

[0012] Preferably, the interactive terminal is specifically used when the corresponding task step flow is obtained by performing step flow generation processing based on the submission information entered by the user on the current task interaction page and the parameter identification agent, the grid planning engine, and the task planning agent: When a user submits a piece of natural language text through the dialog input box on the current task interaction page, the current natural language text is recorded as the first text; the electron microscope image loaded in the electron microscope window on the current task interaction page is used as the current electron microscope image; and the first text is added to the history dialog area. The first text is used as the corresponding scan overview text; the current electron microscope image is identified as empty. If it is, the reference electron microscope image is set to empty; otherwise, the current electron microscope image is further identified as having a Region of Interest (ROI) marker box. If it is, the electron microscope image with the ROI marker box is used as the reference electron microscope image; otherwise, the current electron microscope image is used as the reference electron microscope image. The parameter identification request carrying the scan overview text and the reference electron microscope image is sent to the parameter identification agent. The scan parameter configuration returned by the parameter identification agent is displayed to the current user through a pop-up window, and each parameter of the scan parameter configuration is provided with a modification interface on the pop-up window. After the user clicks the confirmation button on the pop-up window, the scan parameter configuration is updated based on the latest parameter information on the pop-up window, and the pop-up window is closed. The prediction parameter set includes at least the target region parameter, the single-frame field of view size, the magnification range, the overlap rate, the acceleration voltage, the focus type, and the detector type. The system identifies the current electron microscope image. If the current electron microscope image is empty, the corresponding ROI region image is set to empty. If the current electron microscope image is not empty and does not contain the ROI region marker box, the corresponding ROI region image is set as the current electron microscope image. If the current electron microscope image is not empty and contains the ROI region marker box, the marker box sub-image on the current electron microscope image corresponding to the ROI region marker box is used as the corresponding ROI region image. The corresponding ROI region parameters are set to the target region parameters configured in the scanning parameters. The corresponding single-frame field of view size and overlap rate are extracted from the scanning parameters configuration. The grid planning request carrying the ROI region image, ROI region parameters, single-frame field of view size, and overlap rate is sent to the grid planning engine. The system receives the optimized overlap rate and the scanning grid table returned by the grid planning engine. The overlap rate in the scanning parameters configuration is updated based on the optimized overlap rate. The scanning parameters configuration and the scanning grid table are added to the history dialog area. The first text is used as the corresponding planning instruction text; all content in the historical dialogue area is used as the corresponding historical context; the task planning request carrying the planning instruction text and the historical context is sent to the task planning agent; and the task step flow sent back by the task planning agent is sent back.

[0013] Preferably, the interactive terminal is specifically used when the image stitching request is generated based on the cached image of the cache service and sent to the image stitching engine: The electron microscope image loaded in the electron microscope window of the current task interaction page is used as the current electron microscope image; and the current electron microscope image is identified. If the current electron microscope image is empty, the corresponding low magnification panoramic image is set to empty. If the current electron microscope image is not empty and does not have a ROI region marker box, the corresponding low magnification panoramic image is set as the current electron microscope image. If the current electron microscope image is not empty and has the ROI region marker box, the marker box sub-image on the current electron microscope image corresponding to the ROI region marker box is used as the corresponding low magnification panoramic image. The electron microscope images cached in the image cache area corresponding to the current scanning task on the cache service are taken as the corresponding high magnification sub-images, and all the obtained high magnification sub-images are sorted in chronological order to form the corresponding high magnification sub-image sequence; and after obtaining the high magnification sub-image sequence, all cached images in the current image cache area are cleared; The image stitching request, which carries the target area parameters, the scan grid table, the high magnification sub-image sequence, and the low magnification panoramic image, is sent to the image stitching engine.

[0014] Preferably, the interactive terminal is specifically used when the current task interactive page is updated based on the electron microscopy image analysis results sent back by the analytical agent: The charts in the electron microscopy analysis results are displayed via pop-up windows; the scale identification information, various data statistics tables, and various analysis tables in the electron microscopy analysis results are converted into text information and added to the history dialogue area of ​​the current task interaction page; the foreground pixels of the target segmentation mask image in the electron microscopy analysis results are colored, and the resulting colored mask image is superimposed on the electron microscopy image to be analyzed to obtain the corresponding colored electron microscopy image, which is then loaded into the electron microscopy window of the previous task interaction page.

[0015] Preferably, the scheduling service is specifically used when the driver service calls the SEM device corresponding to the device identifier to execute the current task step flow step by step and sends the end notification to the interactive terminal when the current task ends: By querying the locally pre-set device parameter library through the device identifier, the device parameter set of the SEM device corresponding to the device identifier is obtained; wherein, the device parameter library includes multiple device parameter sets, and each device parameter set corresponds to the device identifier of the SEM device; the device parameter set of each SEM device is used to define the instruction code encoding of all device instructions of the current device, define the name, data type, parameter value range constraints of all device instruction parameters of the current device, define whether each device instruction of the current device has prerequisites, and define the prerequisite condition logic judgment rules for device instructions with prerequisites; Then, all the single-step operation nodes of the current task step flow are pushed into the corresponding execution instruction queue in sequence; and the message reading pointer is initially positioned at the first single-step operation node. Each time the message read pointer is positioned, the single-step operation node pointed to by the message read pointer is taken as the current node; the node instruction set of the current node is taken as the current instruction set; and the execution conditions of all instructions in the current instruction set are checked according to the global state information cached in the state cache area corresponding to the current scanning task on the cache service and the current device parameter set. If the check fails, the abnormal alarm information of the global state information cached in the state cache area is set according to the reason for failure, the current task is stopped, and the termination notification specifically set to abnormal termination is sent to the interactive terminal. If the check passes, the current instruction set is forwarded to the SEM device corresponding to the device identifier by calling the standardized interface of the driver service; and a receiving wait is performed after the instruction is forwarded; after the waiting time exceeds the preset standard waiting time, the latest instruction execution feedback added in the execution feedback queue is identified as matching the current node; if they do not match, a preset timeout is set. The operation action is identified. If the timeout operation action is "stop", the current task is stopped and a termination notification specifically set to "abnormal termination" is sent to the interactive terminal. If the timeout operation action is "retransmission", the current instruction set is retransmitted to the current SEM device, and after retransmission, the next round of waiting begins. If a match is found, the system identifies whether there is abnormal feedback in the current instruction execution feedback. If abnormal feedback exists, the system identifies the operation action corresponding to the current abnormal feedback based on a preset abnormal feedback-operation action correspondence rule. If the current operation action is "stop", the current task is stopped and a termination notification specifically set to "abnormal termination" is sent to the interactive terminal. If the current operation action is "rollback", the message reading pointer is positioned to the previous single-step operation node. If there is no abnormal feedback, the system identifies whether the single-step operation node corresponding to the current message reading pointer is the last node. If so, the current task is stopped and a termination notification specifically set to "successful termination" is sent to the interactive terminal. Otherwise, the message reading pointer is positioned to the next single-step operation node.

[0016] This invention provides a natural language-driven SEM scanning task processing system, comprising: a parameter recognition agent, a grid planning engine, a task planning agent, an image stitching engine, an analysis agent, an interactive terminal, a scheduling service, a caching service, a driving service, and an SEM device. The parameter recognition agent performs multimodal feature extraction, joint feature encoding, and scanning parameter prediction based on image and text information. The grid planning engine optimizes the overlap rate based on the ROI region map, ROI region parameters, single-frame field of view size, and overlap rate, and plans the scanning grid based on the optimized overlap rate. The task planning agent uses a thought chain reasoning mechanism to perform multi-step deduction based on natural language text and its corresponding historical context. During the deduction process, it performs atomic operation sequence parsing of the electron microscope equipment operation steps for the current scanning task, generates a task step flow in a specified format based on the parsed sequence, and performs compliance verification on the task step flow. The image stitching engine performs sub-image correction and full-sequence sub-image stitching based on target region parameters, a scanning grid table, a high-magnification sub-image sequence, and a low-magnification panoramic image to obtain a corresponding high-fidelity panoramic image. The analysis agent uses the Uni-AIMS analysis tool to perform scale recognition, particle and pore target segmentation, particle size recognition, pore size recognition, particle distribution analysis, and pore distribution analysis on the electron microscope images to be analyzed. The driver service provides a unified set of standardized interfaces and adapts each standardized interface to the corresponding manufacturer's interface for each SEM device. It also facilitates the intermediate forwarding from upper-layer applications (such as scheduling services and caching services) to lower-layer devices (such as SEM devices) through the standardized interface set. The interaction interface generates a corresponding interactive page (components: dialog input box, history dialog area, status panel, electron microscope window) for the current scan task when the user confirms the creation of a new scan task. Based on the user's submission information and parameters on the current page, it automatically completes parameter settings and task step flow generation using the identification agent, grid planning engine, and task planning agent. Through interaction with the scheduling service, caching service, driver service, and the SEM device's backend, it automatically completes the scan task. At the end of the task, it automatically calls the image stitching engine to stitch the scanned sub-images into a high-fidelity panoramic image and uses the analysis agent to perform online real-time analysis of the high-fidelity panoramic image. Based on the embodiments of the present invention, the problem of manual dependence on parameter setting and task flow customization is solved, the development difficulty and workload of upper-layer applications are simplified, the distortion phenomenon of large field-of-view mosaic images is improved, online real-time analysis of electron micrographs is realized, the task processing difficulty is reduced, the task processing quality is improved, the task processing effect is optimized, and the task processing efficiency is improved. Attached Figure Description

[0017] Figure 1 A module structure diagram of a natural language-driven SEM scanning task processing system provided in an embodiment of the present invention; Figure 2A module structure diagram of a parameter recognition model provided in an embodiment of the present invention; Figure 3 This is a module structure diagram of an overlap rate optimization model provided in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0019] The natural language-driven SEM scanning task processing system provided in this embodiment of the invention, such as... Figure 1 The module structure diagram of a natural language driven SEM scanning task processing system provided in this embodiment of the invention is shown, which mainly includes: parameter recognition agent 1, grid planning engine 2, task planning agent 3, image stitching engine 4, analysis agent 5, interaction terminal 6, scheduling service 7, caching service 8, driving service 9, and SEM device 10.

[0020] The connection relationships of the various components of the system are as follows: the interaction terminal 6 is connected to the parameter recognition agent 1, the grid planning engine 2, the task planning agent 3, the image stitching engine 4, the analysis agent 5, the scheduling service 7, and the cache service 8 respectively; the driver service 9 is connected to the scheduling service 7, the cache service 8, and each SEM device 10 respectively; the scheduling service 7 is also connected to the cache service 8.

[0021] (a) Parameter recognition agent 1: The parameter recognition agent 1 is used to receive parameter recognition requests; and the preset parameter recognition model performs multimodal feature extraction, joint feature encoding and scanning parameter prediction based on the scan overview text and reference electron microscope image carried in the current request to obtain the corresponding predicted parameter set; and substitutes the predicted parameter set into the preset configuration template to generate the corresponding scan parameter configuration and sends it back to the current requester.

[0022] Here, the prediction parameter set of this embodiment of the invention includes at least target region parameters, single-frame field of view size, magnification range, overlap rate, acceleration voltage, focus type, and detector type; the target region parameters consist of the coordinates of the four vertices of the target region; the single-frame field of view size consists of the corresponding field of view width and field of view height; the focus type includes autofocus and manual focus; the detector type includes SE type, BSE type, and EDS type.

[0023] like Figure 2As shown in the module structure diagram of a parameter recognition model provided in an embodiment of the present invention, the model components of the parameter recognition model in this embodiment of the present invention include: a text encoder, an image encoding module, an image encoder, a target detection model, a multimodal feature joint encoding module, and a parameter prediction module.

[0024] like Figure 2 As shown, the first model input of the parameter recognition model is used to receive the scan overview text, the second model input is used to receive the reference electron microscope image, and the model output is used to output the predicted parameter set.

[0025] like Figure 2 As shown, the connection relationships of the model components in the parameter recognition model are as follows: the input end of the text encoder is connected to the input end of the first model, and the output end is connected to the first input end of the multimodal feature joint encoding module; the input end of the image encoding module is connected to the input end of the second model, and the output end is connected to the second input end of the multimodal feature joint encoding module; the image encoding module is also connected to the image encoder and the object detection model respectively; the output end of the multimodal feature joint encoding module is connected to the input end of the parameter prediction module; and the output end of the parameter prediction module is connected to the model output end.

[0026] The functions of each model component in the parameter recognition model are shown below.

[0027] 1) Text encoder: The text encoder in this embodiment of the invention is used to extract text features from the scanned overview text to obtain the corresponding feature vector X1, which is then sent to the multimodal feature joint encoding module. The feature vector X1 has a shape of 1×D1, where D1 is a preset first feature dimension.

[0028] It should be noted that the text encoder in this embodiment of the invention can be implemented based on multiple encoder model structures, including the encoder structure of the VAE model, the BERT series model structure, etc., and can be selected or customized according to application requirements.

[0029] 2) Image encoding module: The image encoding module of this embodiment of the invention is used to identify whether the reference electron microscope image is empty. If the reference electron microscope image is empty, the corresponding feature vectors X2 and X3 are set to all zero vectors. If the reference electron microscope image is not empty, the image encoder is called to perform global feature extraction on the reference electron microscope image to obtain the corresponding feature vector X2. The target detection model is called to perform target detection on the reference electron microscope image with the ROI region bounding box as the detection target to obtain the corresponding target detection box set. The target detection box with the highest confidence in the target detection box set is taken as the first matching box. The module identifies whether the first matching box is empty. If it is, the corresponding feature vector X3 is set to all zero vector. Otherwise, the corresponding feature vector X3 is composed of the center point coordinates, width, and height of the first matching box. The obtained feature vectors X2 and X3 are sent to the multimodal feature joint encoding module.

[0030] In this embodiment, feature vector X2 has a shape of 1×D2, feature vector X3 has a shape of 1×D3, D2 and D3 are the preset second and third feature dimensions, respectively, and D3=4. The image encoder in this embodiment is implemented based on a residual network model; the target detection model in this embodiment is implemented based on the YOLO series model.

[0031] 3) Multimodal feature joint encoding module: The multimodal feature joint encoding module of this invention is used to map feature vectors X1, X2, and X3 to a preset target feature space using three linear layers, denoted as feature vectors Y1, Y2, and Y3 respectively; and to concatenate feature vectors Y1, Y2, and Y3 to obtain the corresponding feature vector Y4; and to use an MLP model to encode feature vector Y4 to obtain the corresponding feature vector Y5, which is then sent to the parameter prediction module.

[0032] The mapping methods for feature vectors Y1, Y2, and Y3, and the encoding method for feature vector Y5 are as follows: , , ; ; W1 and b1 are the weight matrices and bias vectors of the first linear layer. The weight matrix W1 has a shape of D4×D1, and the bias vector b1 has a shape of 1×D4, where D4 is the feature dimension of the target feature space. W2 and b2 are the weight matrices and bias vectors of the second linear layer. The weight matrix W2 has a shape of D4×D2, and the bias vector b2 has a shape of 1×D4. W3 and b3 are the weight matrices and bias vectors of the third linear layer. The weight matrix W3 has a shape of D4×D3, and the bias vector b3 has a shape of 1×D3. The feature vectors Y1, Y2, and Y3 all have a shape of 1×D4. The feature vector Y4 has a shape of 1×3D4. MLP1() is the inference function of the first MLP model.

[0033] 4) Parameter prediction module: The parameter prediction module is used to identify whether the first matching box is empty. If it is, the corresponding target area parameter is set to the default area parameter of the SEM device corresponding to the current scanning task. Otherwise, the coordinates of the four vertices are calculated based on the center point coordinates, width, and height of the first matching box, and the four coordinates are used to form the corresponding target area parameter. The corresponding linear regression prediction head is used to predict the single-frame field size, magnification range, overlap rate, and acceleration voltage. The corresponding classification prediction head is used to predict the focus type and detector type. The corresponding prediction parameter set is composed of the obtained target area parameter, single-frame field size, magnification range, overlap rate, acceleration voltage, focus type, and detector type and output.

[0034] It should be noted that the scanning parameter recognition model in this embodiment of the invention has been trained using a supervised model training method before use. Before model training, a first dataset is constructed through big data collection. The first dataset consists of multiple first data records. Each first data record consists of a corresponding set of scanning overview text, reference electron microscope image, and label parameter set. The label parameter set is labeled by a professional manual or machine annotation system. In the first dataset, some first data records have empty reference electron microscope images, some have non-empty reference electron microscope images without ROI region bounding boxes, and some have non-empty reference electron microscope images with ROI region bounding boxes. During model training, the scanning overview text and reference electron microscope image of each first data record are input into the scanning parameter recognition model to obtain the corresponding prediction parameter set. Each prediction parameter set and its corresponding label parameter set form a corresponding first prediction-label pair. All first prediction-label pairs are input into a preset first model loss function, and the model parameters of the scanning parameter recognition model are optimized in the direction of minimizing the first model loss function. The first model loss function is implemented based on the MSE function.

[0035] It should be noted that the preset configuration template in this embodiment of the invention is a pre-set formatted text template. The formatted text of this template can be customized based on application requirements, and the default format can be TypeScript or JSON.

[0036] (ii) Grid Planning Engine 2: Mesh planning engine 2 receives mesh planning requests and identifies whether the ROI region map carried in the current request is empty. If it is, the overlap rate carried in the current request is used as the corresponding optimized overlap rate. If not, the preset overlap rate optimization model optimizes the overlap rate based on the ROI region map, ROI region parameters, single-frame field of view size, and overlap rate carried in the current request to obtain the corresponding optimized overlap rate. The engine then performs scanning mesh planning based on the optimized overlap rate to obtain the corresponding scanning mesh table and sends the optimized overlap rate and scanning mesh table back to the requester.

[0037] Here, the scan grid table in this embodiment of the invention consists of multiple scan grid records; each scan grid record consists of a single frame field of view size and the corresponding grid center point coordinates. The recording order of the scan grid table can be understood as the grid scan path.

[0038] like Figure 3 As shown in the module structure diagram of an overlap rate optimization model provided in an embodiment of the present invention, the model components of the overlap rate optimization model in this embodiment of the present invention include: a first feature extraction network, a second feature extraction network, a feature fusion layer, a feature encoding layer, an overlap rate prediction layer, and a prediction constraint module.

[0039] like Figure 3 As shown, the first model input of the overlap rate optimization model is used to receive the ROI region map, the second model input is used to receive the ROI region parameters, the single-frame field of view size and the overlap rate, and the model output is used to output the optimized overlap rate.

[0040] like Figure 3 As shown, the connection relationships of the model components in the overlap rate optimization model are as follows: the input of the first feature extraction network is connected to the input of the first model, and its output is connected to the first input of the feature fusion layer; the input of the second feature extraction network is connected to the input of the second model, and its output is connected to the second input of the feature fusion layer; the output of the feature fusion layer is connected to the input of the feature encoding layer; the output of the feature encoding layer is connected to the input of the overlap rate prediction layer; the output of the overlap rate prediction layer is connected to the input of the prediction constraint module; and the output of the prediction constraint module is connected to the model output.

[0041] The model component functions of the overlap rate optimization model are shown below.

[0042] 1) First feature extraction network: The first feature extraction network in this embodiment of the invention is implemented based on a residual network model. The first feature extraction network is used to perform global feature extraction on the ROI region map to generate the corresponding feature vector X. 1 Send to the feature fusion layer.

[0043] 2) Second feature extraction network: The second feature extraction network in this embodiment of the invention is implemented based on an MLP model. The second feature extraction network is used to generate a corresponding feature vector X by performing feature extraction on the initial vector, which is composed of ROI region parameters, single-frame field of view size, and overlap rate. 2 Send to the feature fusion layer.

[0044] 3) Feature fusion layer: The feature fusion layer in this embodiment of the invention is used to perform feature vector X by vector concatenation. 1 and eigenvector X 2 Vector concatenation yields the corresponding feature vector X. 3 Send to the feature coding layer.

[0045] 4) Feature coding layer: The feature encoding layer in this embodiment of the invention is implemented based on an MLP model. The feature encoding layer is used to process the feature vector X. 3 Feature encoding is performed to obtain the corresponding feature vector X. 4 Send to the overlap rate prediction layer.

[0046] 5) Overlap Rate Prediction Layer: The overlap rate prediction layer in this embodiment of the invention is implemented based on a linear regression prediction network. The overlap rate prediction layer is used to predict the overlap rate based on the feature vector X. 4 The corresponding predicted overlap rate ρ is obtained by performing overlap rate prediction. * Send to the prediction constraint module.

[0047] 6) Prediction Constraint Module: The prediction constraint module in this embodiment of the invention denotes the overlap rate of the model input as ρ. old And based on the overlap rate ρ old And the preset percentage α versus the predicted overlap rate ρ * The corresponding optimized overlap rate ρ is obtained by applying constraints. new : ; Where 0 < α < 1. Clip() is the Clip truncation function.

[0048] It should be noted that the overlap rate optimization model in this embodiment of the invention has been trained using a supervised model training method before use. Before model training, a second dataset is constructed through big data collection. The second dataset consists of multiple second data records. Each second data record consists of a corresponding ROI region map, ROI region parameters, single-frame field of view size, overlap rate, and label overlap rate. The label overlap rate is labeled by a professional manual or machine labeling system and ensured to be within a specified range [overlap rate × (1-α), overlap rate × (1+α)]. During model training, the ROI region map, ROI region parameters, single-frame field of view size, and overlap rate of each second data record are input into the overlap rate optimization model to obtain the corresponding predicted overlap rate. Each predicted overlap rate and its corresponding label overlap rate form a corresponding second prediction-label pair. All second prediction-label pairs are input into a preset second model loss function, and the model parameters of the overlap rate optimization model are optimized in the direction that minimizes the second model loss function. The second model loss function is implemented based on the MSE function.

[0049] In one specific implementation of this invention, the grid planning engine 2 is specifically used to obtain the corresponding scan grid table by performing scan grid planning based on the optimized overlap rate: Based on the ROI region parameters, single-frame field of view size, and optimized overlap rate, the center point coordinates of each scan grid in each grid row of the ROI region are calculated to obtain the corresponding grid center point coordinates; and a corresponding scan grid record is formed by the center point coordinates of each grid and the single-frame field of view size; and all scan grid records are sorted according to the scanning order to form the corresponding scan grid table.

[0050] (III) Task Planning Agent 3: Task planning agent 3 receives task planning requests; inputs the planning instruction text carried in the current request and its corresponding historical context into a preset prompt instruction template to generate corresponding task planning instructions; inputs the task planning instructions into a preset scanning task planning model, which performs multi-step deduction based on the instruction prompts and the thought chain reasoning mechanism, with reference to the historical context, and performs atomic operation sequence parsing of the electron microscope equipment operation steps of the current scanning task during the deduction process, and generates a task step flow in a specified format based on the parsed sequence; performs compliance verification on the task step flow; and sends back the verified task step flow to the current requester.

[0051] Here, the scanning task planning model of this invention is based on a class of large language models or multimodal large models, including at least the GPT series models, DeepSeek series models, SenseNova-MARS series models, Wenxin series models, and Tongyi Qianwen Qwen series models.

[0052] It should be noted that the scanning task planning model in this embodiment of the invention has completed large-scale model pre-training and targeted fine-tuning before use. Before targeted fine-tuning of the model, a third dataset is constructed through big data collection. The third dataset consists of multiple third data records. Each third data record consists of a set of corresponding planning instruction text, historical context, and corresponding label thinking chain and label step flow. The label thinking chain and label step flow are labeled by a professional manual or machine labeling system. During targeted fine-tuning, the planning instruction text and historical context of each third data record are input into the large model prompt instruction template to generate corresponding task planning instructions. These task planning instructions are then input into the scanning task planning model for processing to obtain corresponding derivation thought chains and task step flows. Each derivation thought chain and its corresponding label thought chain forms a corresponding third prediction-label pair, and each task step flow and its corresponding label step flow forms a corresponding fourth prediction-label pair. All third prediction-label pairs are input into a preset third model loss function, and all fourth prediction-label pairs are input into a preset fourth model loss function. The third and fourth model loss functions are then added together to obtain the corresponding overall loss function. The model parameters of the scanning task planning model are then fine-tuned in the direction that minimizes the overall loss function. Both the third and fourth model loss functions are based on the negative log-likelihood loss function.

[0053] The task step flow of this embodiment of the invention is formed by sequentially sorting multiple single-step operation nodes; each single-step operation node includes a node identifier, a node type, and a node instruction set; the node type includes a single-instruction node and a multi-instruction node; the node instruction set consists of one or more first instructions; the first instruction includes an instruction code and instruction parameters; when the node type is a single-instruction node, the total number of instructions in the node instruction set is 1; when the node type is a multi-instruction node, the total number of instructions in the node instruction set is greater than 1.

[0054] The key paragraphs of the prompt instruction template in this embodiment of the invention include at least five parts: model role description text, historical dialogue text, user request text, inference chain request text, and output format constraint text. Among these, the model role description text, inference chain request text, output format constraint text, and step instruction constraint text are each a fixed piece of natural language text. The historical dialogue text and user request text are configurable text sections of the template.

[0055] The model role description text is used to set the task role of the scanning task planning model as the SEM intelligent control assistant; and to describe the corresponding role task of the model: using historical dialogue text as context reference, and according to the reasoning chain requirement text set by the reasoning chain, to perform multi-step deduction according to the user requirement text, and to perform atomic operation sequence parsing of electron microscope equipment operation steps during the deduction process, and to generate a task step flow in the format specified by the output format constraint text based on the parsed sequence.

[0056] The historical dialogue text is configured based on the historical context input from the template.

[0057] The user requires the text segment to be configured based on the planning instruction text input from the template.

[0058] The reasoning chain requirement is to provide a step-by-step explanation of the reasoning steps in the thought chain of the scanning task planning model.

[0059] The output format constraint section is used to describe the formatted text format of the task step flow output by the scanning task planning model. The formatted text format includes at least TypeScript format and JSON format.

[0060] In another specific implementation of this invention, the task planning agent 3 is specifically used when performing compliance checks on the task step flow: Step A1: Perform a sequential traversal of all single-step operation nodes in the task step flow; during this traversal, the currently traversed single-step operation node is designated as the current node; poll each first instruction of the current node; during this polling, the first instruction polled is designated as the current instruction; and verify the compliance of the instruction code and instruction parameters of the current instruction based on a preset instruction rule set; if the current instruction is successfully verified, proceed to the next first instruction and continue polling until all first instructions of the current node have been polled; if the current instruction fails to be verified, end this round of polling and traversal, and set the corresponding step flow verification result to failure; after all polling in this round is successfully verified, proceed to the next single-step operation node and continue traversing until all single-step operation nodes have been traversed, and set the corresponding step flow verification result to success when all instructions of all single-step operation nodes have been successfully verified. The instruction rule set includes multiple instruction rules; each instruction rule corresponds one-to-one with an instruction code; each instruction rule is used to specify the instruction code encoding of its corresponding instruction, and to specify the name, data type, parameter value range constraints of each instruction parameter of its corresponding instruction, and to specify whether its corresponding instruction has preconditions, and to specify the logical judgment rules corresponding to the preconditions when its corresponding instruction has preconditions. Step A2, and identify whether the obtained step flow verification result is successful; if yes, the compliance check is confirmed to have passed; if no, the compliance check is confirmed to have failed.

[0061] (iv) Image stitching engine 4: Image stitching engine 4 is used to receive image stitching requests; and based on the target area parameters, scan grid table, high magnification sub-image sequence and low magnification panoramic image carried in the current request, it performs sub-image correction and full sequence sub-image stitching to obtain the corresponding high-fidelity panoramic image and sends it back to the current requester.

[0062] Here, the high-magnification sub-image sequence of this embodiment consists of multiple high-magnification sub-images; each high-magnification sub-image in the high-magnification sub-image sequence corresponds one-to-one with the scanning grid record in the scanning grid table, and each high-magnification sub-image is a grid electron microscope scan of a corresponding scanning grid in the target region corresponding to the target region parameter. When the low-magnification panoramic image of this embodiment is not empty, it is an electron microscope scan of the entire target region corresponding to the target region parameter. The scanning magnification of the high-magnification sub-images in this embodiment is greater than that of the low-magnification panoramic image.

[0063] In another specific implementation of this invention, the image stitching engine 4 is specifically used to send back a corresponding high-fidelity panoramic image to the current requester when performing sub-image correction and full-sequence sub-image stitching based on the target area parameters, scan grid table, high-magnification sub-image sequence, and low-magnification panoramic image carried in the current request: Step B1: When the low magnification panoramic image is not empty, based on the target area parameters and the scanning grid table, the corresponding low magnification sub-images of each high magnification sub-image sequence in the low magnification panoramic image are confirmed, and distortion correction is performed on the corresponding high magnification sub-images based on the feature point matching relationship of each low magnification sub-image. Specifically, this includes: Step B11, taking each high-magnification sub-image in the high-magnification sub-image sequence as the current high-magnification sub-image; taking the magnification corresponding to the current high-magnification sub-image as the current high-magnification; taking the magnification corresponding to the low-magnification panoramic image as the current low-magnification; taking the scan grid record corresponding to the current sub-image in the scan grid table as the current record; confirming the scan grid area corresponding to the current record on the low-magnification panoramic image based on the target area parameters and the single-frame field of view size and grid center point coordinates of the current record; and extracting the sub-image of the current scan grid area as the current low-magnification sub-image. Step B12 involves performing feature point detection on the current high-magnification sub-image and the current low-magnification sub-image respectively using a preset feature point detection algorithm to obtain the corresponding feature point sets for the high-magnification sub-image and the low-magnification sub-image; confirming all matching feature point pairs in the high-magnification sub-image and the low-magnification sub-image feature point sets based on a preset feature point matching algorithm; identifying the projection transformation relationship between the current high-magnification sub-image and the current low-magnification sub-image based on all obtained matching feature point pairs; and performing pixel coordinate transformation on the current high-magnification sub-image based on the identified projection transformation relationship. Among them, the feature point detection algorithms include at least the SIFT algorithm, the SURF algorithm, and the ORB algorithm; the feature point matching algorithms include at least the Brute-Force algorithm and the FLANN algorithm; Step B2: Construct a scanning grid plane space, and identify all adjacent sub-image pairs of the high magnification sub-image sequence in the scanning grid plane space to obtain the corresponding set of adjacent sub-image pairs; Specifically, this includes: constructing a scanning grid plane space based on target region parameters and known single-frame field of view size and overlap rate; retrieving all spatially adjacent sub-images of each high-magnification sub-image sequence based on the scanning grid plane space; forming a corresponding adjacent sub-image pair by combining each high-magnification sub-image with its corresponding adjacent sub-image; deduplicating all obtained adjacent sub-image pairs; and forming a corresponding adjacent sub-image pair set by combining all deduplicated adjacent sub-image pairs. The scanning grid plane space includes multiple scanning grids, and the height and width of each scanning grid satisfy the single-frame field of view size; the overlap ratio of any two adjacent scanning grids in the plane space satisfies the overlap rate; each scanning grid corresponds to a high magnification sub-image; all spatially adjacent sub-images include the left neighbor, right neighbor, upper neighbor, and lower neighbor of the current sub-image; each adjacent sub-image pair in the set of adjacent sub-image pairs is denoted as the corresponding sub-image pair Z. i 1 ≤ index i ≤ N, where N is the total number of subgraph pairs in the current subgraph pair set; each subgraph pair Z i The two high-magnification subgraphs are denoted as the corresponding subgraph P. i,1 P i,2 ; Step B3: Calculate the Z pairs for each subgraph based on the phase correlation method. i The corresponding relative displacement s i And from all the relative displacements s obtained i Form the corresponding set of relative displacements; Specifically, this includes: aligning each subgraph with Z... i As the current subgraph pair; and for the subgraph P of the current subgraph pair i,1 P i,2 Performing a Fourier transform yields the corresponding spectrum F. i,1Spectrum F i,2 ; and based on the spectrum F i,1 and spectrum F i,2 F i,2 Calculate the corresponding cross-power spectrum. , For the spectrum F i,2 The complex conjugate of ε is a preset small constant to prevent the denominator from being zero; and the inverse Fourier transform of the cross-power spectrum R is performed to obtain the corresponding correlation function r(x,y), where x and y are the sub-graphs P. i,1 P i,2 The lateral and longitudinal relative displacements; and the extreme points (x,y) of the related function r(x,y). peak ,y peak Solve the problem; and calculate the lateral relative displacement x of the extreme points obtained from the solution. peak Longitudinal relative displacement y peak Let x be the corresponding lateral relative displacement. i Longitudinal relative displacement y i ; and determined by the lateral relative displacement x i and longitudinal relative displacement y i The corresponding relative displacement s i ; and from all the relative displacements s obtained i Form the corresponding set of relative displacements; Step B4, pair each subgraph with Z i Subgraph P i,1 P i,2 The coordinates of the corresponding grid center point are marked as the center point c of the corresponding subgraph. i,1 c i,2 ; and based on all subgraphs, Z i The corresponding subgraph center point c i,1 c i,2 and relative displacement s i Construct a translation optimization objective function E1; and based on a preset first optimization algorithm, determine the subgraph center point offset Δc that minimizes the translation optimization objective function E1. i,1 , △c i,2 An estimate is performed, and the coordinates of the grid center points of each high-magnification sub-image are reset based on the estimate results; The translation optimization objective function E1 is: ; The first optimization algorithm includes the least squares method and the simulated annealing algorithm; It should be noted that during the optimization process, the offset Δc of the center point of the sub-image corresponding to the same high magnification sub-image i,1 or △c i,2 Maintaining global uniqueness; after optimization, due to the offset Δc of the center point of the sub-image corresponding to the same high magnification sub-image.i,1 or △c i,2 To maintain global uniqueness, the offset of the center point of each high-magnification sub-image is recorded as △c. Thus, the coordinates of the center point of the grid after the reset of each high-magnification sub-image = the coordinates of the center point of the grid before the reset + △c. Step B5: Apply the feature point detection algorithm to each sub-image pair Z. i Feature point detection is performed on the two subgraphs to obtain the corresponding first and second point sets; and all matching point pairs in the first and second point sets are confirmed based on the feature point matching algorithm to obtain the corresponding set of matching point pairs D. i ; Wherein, the set of matching point pairs D i subgraph pair Z i One-to-one correspondence; matching point pair set D i Includes multiple matching point pairs d i,j 1 ≤ index j ≤ M, where M is the total number of matching point pairs in the current matching point pair set; each matching point pair d i,j From the coordinates of point p i,j,1 Point coordinates p i,j,2 Composition; point coordinates p i,j,1 For the corresponding subgraph P i,1 The coordinates of point p i,j,2 For the corresponding subgraph P i,2 The coordinates of the point; Step B6: Initialize a corresponding linear transformation function for each high-magnification sub-image in the high-magnification sub-image sequence as the initial projection transformation relationship T. u 1 ≤ index u ≤ K, where K is the total number of subgraphs in the current subgraph sequence; and based on all projection transformation relationships T u and the set of all matching point pairs D i Construct a nonlinear distortion optimization objective function E2; and based on a pre-set second optimization algorithm, perform all projection transformation relationships T that minimize the nonlinear distortion optimization objective function E2. u Make an estimate; Among them, the various projection transformation relationships T u Used to perform coordinate projection of the scan grid plane space onto its corresponding high magnification sub-image; The objective function E2 for nonlinear distortion optimization is: ; Let p be the coordinates of the point. i,j,1 Corresponding subgraph P i,1 The corresponding projection transformation relationship T u , Let p be the coordinates of the point. i,j,1 Through projection transformation relationship The coordinates of the projection point obtained after calculation; Let p be the coordinates of the point. i,j,2 Corresponding subgraph P i,2 The corresponding projection transformation relationship T u , Let p be the coordinates of the point. i,j,2 Through projection transformation relationship The coordinates of the projection point obtained after calculation; The second optimization algorithm includes the Levenberg-Marquardt algorithm, the Gauss-Newton algorithm, and the simulated annealing algorithm; Step B7, based on the estimated projection transformation relationships T u The pixel coordinates of the corresponding high-magnification sub-image are transformed by scanning the grid plane space to obtain the corresponding grid plane sub-image; and the obtained grid plane sub-images are stitched together in spatial arrangement to obtain the corresponding high-fidelity panoramic image, which is then sent back to the current requester.

[0064] (v) Analysis of Agent 5: Analysis agent 5 is used to receive graph analysis requests; and input the electron microscope image to be analyzed carried in the current request into the preset Uni-AIMS analysis tool for processing to obtain the corresponding electron microscope image analysis results, which are then sent back to the current requester.

[0065] The Uni-AIMS analysis tool in this invention is a software tool for depth analysis of electron micrographs. This Uni-AIMS analysis tool is used to perform scale identification, particle and pore target segmentation, particle size identification, pore diameter identification, particle distribution analysis, and pore distribution analysis on the electron micrograph to be analyzed. The electron micrograph analysis results output by the Uni-AIMS analysis tool include at least scale identification information, target segmentation mask image, statistical table of particle size / pore diameter data, and analysis chart of particle / pore distribution.

[0066] It should be noted that the analytical agent 5 in this embodiment of the invention can also generate corresponding standard reports based on the electron microscopy image analysis results output by the Uni-AIMS analysis tool, according to international, national or industry standards in fields such as electron microscopy characterization and material testing. Such reports can be plain text reports or multimodal reports with multimodal information such as graphs and tables.

[0067] (vi) Interactive Terminal 6: The interaction terminal 6 is used to generate a corresponding task interaction page 61 for the current scanning task when the user confirms the creation of a new scanning task for a certain SEM device 10 through the client menu. The components of the task interaction page 61 include at least a dialog input box, a history dialog area, a status panel, and an electron microscope window. Based on the submission information entered by the user on the current task interaction page 61 and the parameter recognition agent 1, grid planning engine 2, and task planning agent 3, the corresponding task step flow is generated. The device identifier of the current SEM device 10 and the task step flow are sent to the scheduling service 7. Before receiving the end notification sent by the scheduling service 7, the status panel is updated according to the global status information pushed by the caching service 8. After receiving the end notification, an image stitching request is generated based on the cached image of the caching service 8 and sent to the image stitching engine 4. The high-fidelity panoramic image returned by the image stitching engine 4 is used as the electron microscope image to be analyzed. The image analysis request carrying the electron microscope image to be analyzed is sent to the analysis agent 5. The current task interaction page 61 is updated based on the electron microscope image analysis results returned by the analysis agent 5.

[0068] It should be noted that the layout style of the task interaction page 61 in this embodiment of the invention adopts the Integrated Development Environment (IDE) layout style. The history dialogue area on the task interaction page 61 is used to display the human-computer interaction text content of the current scanning task one by one. The status panel on the task interaction page 61 is used to display the global status information of the current scanning task. The global status information in this embodiment of the invention includes at least scanning progress, vacuum status, electron beam parameters, detector status, and abnormal alarm information. The electron microscope window on the task interaction page 61 is used to load electron microscope images, and supports loading via file selection components or manual drag and drop, and supports manually drawing to select ROI regions on the electron microscope images loaded in the window.

[0069] In another specific implementation of this invention, the interactive terminal 6 is specifically used to generate the corresponding task step flow based on the submission information and parameters entered by the user on the current task interaction page 61, as well as the step flow generation processing of the intelligent agent 1, the grid planning engine 2, and the task planning intelligent agent 3: Step C1: When the user submits a piece of natural language text through the dialog input box of the current task interaction page 61, the current natural language text is recorded as the first text; the electron microscope image loaded in the electron microscope window of the current task interaction page 61 is used as the current electron microscope image; and the first text is added to the history dialog area. Step C2: The first text is used as the corresponding scan overview text; the current electron microscope image is checked to see if it is empty. If it is, the reference electron microscope image is set to empty. Otherwise, the current electron microscope image is checked to see if it has a ROI region marker box. If it does, the electron microscope image with the ROI region marker box is used as the reference electron microscope image. Otherwise, the current electron microscope image is used as the reference electron microscope image. The parameter recognition request carrying the scan overview text and the reference electron microscope image is sent to parameter recognition agent 1. The scan parameter configuration returned by parameter recognition agent 1 is displayed to the current user through a pop-up window. Each parameter of the scan parameter configuration is provided on the pop-up window, and the scan parameter configuration is updated based on the latest parameter information on the pop-up window after the user clicks the confirmation button on the pop-up window. The pop-up window is then closed. Here, the prediction parameter set includes at least the target area parameters, single-frame field of view size, magnification range, overlap rate, acceleration voltage, focus type, and detector type; Step C3 involves identifying the current electron microscope image. If the current electron microscope image is empty, the corresponding ROI region image is set to empty. If the current electron microscope image is not empty and does not have an ROI region marker box, the corresponding ROI region image is set as the current electron microscope image. If the current electron microscope image is not empty and has an ROI region marker box, the marker box sub-image corresponding to the ROI region marker box on the current electron microscope image is used as the corresponding ROI region image. The corresponding ROI region parameters are set to the target region parameters configured in the scan parameters. The corresponding single-frame field of view size and overlap rate are extracted from the scan parameters configuration. A grid planning request carrying the ROI region image, ROI region parameters, single-frame field of view size, and overlap rate is sent to the grid planning engine 2. The optimized overlap rate and scan grid table returned by the grid planning engine 2 are received. The overlap rate in the scan parameters configuration is updated based on the optimized overlap rate. The scan parameters configuration and scan grid table are added to the history dialog area. Step C4, and take the first text as the corresponding planning instruction text; take all the contents of the historical dialogue area as the corresponding historical context; send the task planning request carrying the planning instruction text and historical context to the task planning agent 3; and send the task step flow back from the task planning agent 3.

[0070] In another specific implementation of this invention, the interactive terminal 6 is specifically used to send an image stitching request to the image stitching engine 4 based on the cached image of the cache service 8: Step D1: Take the electron microscope image loaded in the electron microscope window of the current task interaction page 61 as the current electron microscope image; and identify the current electron microscope image. If the current electron microscope image is empty, set the corresponding low magnification panoramic image to be empty. If the current electron microscope image is not empty and does not have a ROI region marker box, set the corresponding low magnification panoramic image as the current electron microscope image. If the current electron microscope image is not empty and has a ROI region marker box, take the marker box sub-image on the current electron microscope image corresponding to the ROI region marker box as the corresponding low magnification panoramic image. Step D2: The electron microscope images cached in the image cache area 84 corresponding to the current scanning task on the cache service 8 are taken as the corresponding high magnification sub-images, and all the obtained high magnification sub-images are sorted in chronological order to form the corresponding high magnification sub-image sequence; after obtaining the high magnification sub-image sequence, all cached images in the current image cache area 84 are cleared. Step D3 involves sending an image stitching request, carrying target area parameters, a scan grid table, a high-magnification sub-image sequence, and a low-magnification panoramic image, to the image stitching engine 4.

[0071] In another specific implementation of this invention, the interactive terminal 6 is specifically used to update the current task interactive page 61 based on the electron microscopy image analysis results sent back by the analytical agent 5: The charts in the electron microscopy analysis results are displayed via pop-up windows; the scale recognition information, various statistical tables, and various analysis tables in the electron microscopy analysis results are converted into text information and added to the history dialogue area of ​​the current task interaction page 61; the foreground pixels of the target segmentation mask image in the electron microscopy analysis results are colored, and the obtained color mask image is superimposed on the electron microscopy image to be analyzed to obtain the corresponding color electron microscopy image, and the color electron microscopy image is loaded into the electron microscopy window of the previous task interaction page 61.

[0072] (vii) Dispatch Service 7: After receiving the device identifier and task step flow, the scheduling service 7 creates a set of corresponding execution instruction queues 81, execution feedback queues 82, status buffers 83 and image buffers 84 for the current scanning task on the cache service 8; and calls the SEM device 10 corresponding to the device identifier through the driver service 9 to execute the current task step flow step by step and sends an end notification to the interactive terminal 6 when the current task ends.

[0073] In another specific implementation of this invention, the scheduling service 7 is specifically used to execute the current task step flow step by step by calling the SEM device 10 corresponding to the device identifier through the driver service 9 and sending an end notification to the interaction terminal 6 when the current task ends: Step E1: Query the local preset device parameter library through the device identifier to obtain the device parameter set of SEM device 10 corresponding to the device identifier; The device parameter library includes multiple device parameter sets, each corresponding to a device identifier of a SEM device 10. The device parameter sets of each SEM device 10 are used to define the instruction code encoding of all device instructions of the current device, define the name, data type, parameter value range constraints of all device instruction parameters of the current device, define whether each device instruction of the current device has prerequisite conditions, and define the prerequisite condition logic judgment rules for device instructions with prerequisite conditions. Step E2, and push all single-step operation nodes of the current task step flow into the corresponding execution instruction queue 81 in batches according to their order; and initially position the message reading pointer to the first single-step operation node; Step E3: Each time the message read pointer is positioned, the single-step operation node pointed to by the message read pointer is taken as the current node; the node instruction set of the current node is taken as the current instruction set; and the execution conditions of all instructions in the current instruction set are checked according to the global state information cached in the state cache area 83 corresponding to the current scanning task on the cache service 8 and the current device parameter set. If the check fails, the abnormal alarm information of the global state information cached in the state cache area 83 is set according to the reason for failure, the current task is stopped, and a specific termination notification of abnormal termination is sent to the interactive terminal 6. If the check passes, the current instruction set is forwarded to the SEM device 10 corresponding to the device identifier by calling the standardized interface of the driver service 9; and the receiving wait is performed after the instruction is forwarded; after the waiting time exceeds the preset standard waiting time, the execution feedback of the latest added instruction in the execution feedback queue 82 is identified to match the current node; if they do not match, the preset... Timeout actions are identified. If the timeout action is "stop," the current task is stopped, and a notification indicating an abnormal termination is sent to the interaction terminal 6. If the timeout action is "retransmission," the current instruction set is retransmitted to the current SEM device 10, and the process enters the next round of waiting. If a match is found, the system checks if there is any abnormal feedback in the current instruction execution feedback. If abnormal feedback exists, the system identifies the corresponding action based on the preset abnormal feedback-action correspondence rules. If the current action is "stop," the current task is stopped, and a notification indicating an abnormal termination is sent to the interaction terminal 6. If the current action is "rollback," the message reading pointer is positioned to the previous single-step operation node. If there is no abnormal feedback, the system checks if the single-step operation node corresponding to the current message reading pointer is the last node. If so, the current task is stopped, and a notification indicating a successful termination is sent to the interaction terminal 6. Otherwise, the message reading pointer is positioned to the next single-step operation node.

[0074] Here, the standard waiting time in this embodiment of the invention is a pre-set time length parameter. The timeout operation in this embodiment of the invention is a pre-set operation action type, including two types: stop and resend. The exception feedback-operation action correspondence rule in this embodiment of the invention is a pre-set correspondence rule, including multiple exception feedback-operation action correspondence records. Each exception feedback-operation action correspondence record consists of a type of exception feedback and its corresponding operation action, the operation action type including stop and rollback.

[0075] (viii) Caching Service 8: The caching service 8 is used to store the corresponding execution instruction queue 81, execution feedback queue 82, status cache 83 and image cache 84 for all scanning tasks; and pushes the global status information of each status cache 83 to the corresponding interactive terminal 6 at a preset push frequency.

[0076] Here, the push frequency in this embodiment of the invention is a pre-set time frequency parameter.

[0077] The execution instruction queue 81 of this embodiment of the invention is used to sequentially store all single-step operation nodes of the task step flow, with each queue message corresponding to a single-step operation node; the execution instruction queue 81 has a configurable message read pointer for locating the current execution node.

[0078] The execution feedback queue 82 in this embodiment of the invention is used to store the instruction execution feedback corresponding to each single-step operation node; the instruction execution feedback includes a node identifier, a node type, and an instruction feedback set; the instruction feedback set consists of one or more first instruction feedbacks; the first instruction feedback includes a first instruction and a first feedback; when the node type is a single instruction node, the total number of instruction feedbacks in the instruction feedback set is 1; when the node type is a multi-instruction node, the total number of instruction feedbacks in the instruction feedback set is greater than 1.

[0079] The state cache area 83 in this embodiment of the invention is used to store global state information.

[0080] In this embodiment of the invention, the image buffer 84 is used to store electron microscope images. It should be noted that when a normal task is executed successfully, the total number of electron microscope images cached in the image buffer 84 should match the total number of records in the scan grid table.

[0081] (ix) Driver Service 9: The driver service 9 provides a unified set of standardized interfaces and adapts each standardized interface to the corresponding manufacturer's equipment interface of each SEM device 10; it forwards the instruction information sent by the scheduling service 7 to the designated SEM device 10 through the standardized interface set; and it writes the instruction execution feedback, global status information or electron microscope images sent by each SEM device 10 into the corresponding execution feedback queue 82, status buffer 83 or image buffer 84 through the standardized interface set.

[0082] (x) SEM equipment 10: SEM device 10 is used to perform single or multiple instruction operations based on received instruction information to obtain corresponding instruction execution feedback, and write the instruction execution feedback into the corresponding execution feedback queue 82 through driver service 9; and update the global status information of the current scanning task according to a preset status update frequency, and write the latest global status information into the corresponding status buffer 83 through driver service 9; and after each grid cell scan is completed and the corresponding electron microscope image is generated, write the latest electron microscope image into the corresponding image buffer 84 through driver service 9.

[0083] Here, the state update frequency in this embodiment of the invention is a pre-set time frequency parameter.

[0084] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the foregoing method embodiments are generated. The computer described above can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The aforementioned computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the aforementioned computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, Bluetooth, microwave, etc.) means. The aforementioned computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0085] This invention provides a natural language-driven SEM scanning task processing system, comprising: a parameter recognition agent, a grid planning engine, a task planning agent, an image stitching engine, an analysis agent, an interactive terminal, a scheduling service, a caching service, a driving service, and an SEM device. The parameter recognition agent performs multimodal feature extraction, joint feature encoding, and scanning parameter prediction based on image and text information. The grid planning engine optimizes the overlap rate based on the ROI region map, ROI region parameters, single-frame field of view size, and overlap rate, and plans the scanning grid based on the optimized overlap rate. The task planning agent uses a thought chain reasoning mechanism to perform multi-step deduction based on natural language text and its corresponding historical context. During the deduction process, it performs atomic operation sequence parsing of the electron microscope equipment operation steps for the current scanning task, generates a task step flow in a specified format based on the parsed sequence, and performs compliance verification on the task step flow. The image stitching engine performs sub-image correction and full-sequence sub-image stitching based on target region parameters, a scanning grid table, a high-magnification sub-image sequence, and a low-magnification panoramic image to obtain a corresponding high-fidelity panoramic image. The analysis agent uses the Uni-AIMS analysis tool to perform scale recognition, particle and pore target segmentation, particle size recognition, pore size recognition, particle distribution analysis, and pore distribution analysis on the electron microscope images to be analyzed. The driver service provides a unified set of standardized interfaces and adapts each standardized interface to the corresponding manufacturer's interface for each SEM device. It also facilitates the intermediate forwarding from upper-layer applications (such as scheduling services and caching services) to lower-layer devices (such as SEM devices) through the standardized interface set. The interaction interface generates a corresponding interactive page (components: dialog input box, history dialog area, status panel, electron microscope window) for the current scan task when the user confirms the creation of a new scan task. Based on the user's submission information and parameters on the current page, it automatically completes parameter settings and task step flow generation using the identification agent, grid planning engine, and task planning agent. Through interaction with the scheduling service, caching service, driver service, and the SEM device's backend, it automatically completes the scan task. At the end of the task, it automatically calls the image stitching engine to stitch the scanned sub-images into a high-fidelity panoramic image and uses the analysis agent to perform online real-time analysis of the high-fidelity panoramic image. Based on the embodiments of the present invention, the problem of manual dependence on parameter setting and task flow customization is solved, the development difficulty and workload of upper-layer applications are simplified, the distortion phenomenon of large field-of-view mosaic images is improved, online real-time analysis of electron micrographs is realized, the task processing difficulty is reduced, the task processing quality is improved, the task processing effect is optimized, and the task processing efficiency is improved.

[0086] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0087] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0088] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A natural language-driven SEM scanning task processing system, characterized in that, The system includes: a parameter recognition agent, a grid planning engine, a task planning agent, an image stitching engine, an analysis agent, an interactive terminal, a scheduling service, a caching service, a driver service, and an SEM device; The interactive terminal is connected to the parameter recognition agent, the grid planning engine, the task planning agent, the image stitching engine, the analysis agent, the scheduling service, and the caching service, respectively; the driving service is connected to the scheduling service, the caching service, and each of the SEM devices, respectively; the scheduling service is also connected to the caching service. The parameter recognition agent is used to receive parameter recognition requests; and the preset parameter recognition model performs multimodal feature extraction, joint feature encoding, and scanning parameter prediction based on the scan overview text and reference electron microscope image carried in the current request to obtain the corresponding predicted parameter set; and substitutes the predicted parameter set into the preset configuration template to generate the corresponding scan parameter configuration and sends it back to the current requester; The grid planning engine receives grid planning requests and identifies whether the ROI region map carried in the current request is empty. If it is, the overlap rate carried in the current request is used as the corresponding optimized overlap rate. If not, the preset overlap rate optimization model optimizes the overlap rate based on the ROI region map, ROI region parameters, single-frame field of view size, and overlap rate carried in the current request to obtain the corresponding optimized overlap rate. The engine then performs scanning grid planning based on the optimized overlap rate to obtain the corresponding scanning grid table and sends the optimized overlap rate and the scanning grid table back to the current requester. The task planning agent receives task planning requests and inputs the planning instruction text carried in the current request and its corresponding historical context into a preset prompt instruction template to generate corresponding task planning instructions. The task planning instructions are then input into a preset scanning task planning model. The model, based on the instruction prompts and a thought chain reasoning mechanism, uses the historical context as a reference to perform multi-step deduction based on the planning instruction text. During the deduction process, atomic operation sequence parsing is performed on the electron microscope equipment operation steps of the current scanning task, and a task step flow in a specified format is generated based on the parsed sequence. The task step flow undergoes compliance verification, and the verified task step flow is sent back to the current requester. The image stitching engine is used to receive image stitching requests; and to perform sub-image correction and full-sequence sub-image stitching based on the target area parameters carried in the current request, the scanning grid table, the high magnification sub-image sequence, and the low magnification panoramic image to obtain the corresponding high-fidelity panoramic image and send it back to the current requester; The analytical agent is used to receive graph analysis requests; and input the electron microscope image to be analyzed carried in the current request into a preset Uni-AIMS analysis tool for processing to obtain the corresponding electron microscope image analysis results, which are then sent back to the current requester. The interactive terminal is used to generate a corresponding task interaction page for the current scanning task when the user confirms the creation of a new scanning task for a certain SEM device through the client menu. The components of the task interaction page include at least a dialog input box, a history dialog area, a status panel, and an electron microscope window. Based on the submission information entered by the user on the current task interaction page, and the parameter recognition agent, the grid planning engine, and the task planning agent, a step flow generation process is performed to obtain the corresponding task step flow. The device identifier of the current SEM device and the task step flow are sent to the scheduling service. Before receiving the end notification from the scheduling service, the status panel is updated based on the global status information pushed by the caching service. After receiving the end notification, an image stitching request is generated based on the cached image from the caching service and sent to the image stitching engine. The high-fidelity panoramic image returned by the image stitching engine is used as the electron microscope image to be analyzed. The image analysis request carrying the electron microscope image to be analyzed is sent to the analysis agent. The current task interaction page is updated based on the electron microscope image analysis results returned by the analysis agent. Upon receiving the device identifier and the task step flow, the scheduling service creates a set of corresponding execution instruction queues, execution feedback queues, status buffers, and image buffers for the current scanning task on the caching service; and calls the SEM device corresponding to the device identifier through the driving service to execute the current task step flow step by step, and sends the end notification to the interactive terminal when the current task ends. The caching service is used to store the corresponding execution instruction queue, execution feedback queue, status cache area, and image cache area for all scanning tasks; and to push the global status information of each status cache area to the corresponding interactive terminal at a preset push frequency. The driving service provides a unified set of standardized interfaces and adapts each standardized interface to the corresponding manufacturer's equipment interface of each SEM device. It also forwards the instruction information sent by the scheduling service to the designated SEM device through the standardized interface set and writes the instruction execution feedback, global status information or electron microscope image sent by each SEM device into the corresponding execution feedback queue, status cache or image cache through the standardized interface set. The SEM device is used to perform single or multiple instruction operations based on the received instruction information to obtain corresponding instruction execution feedback, and write the instruction execution feedback into the corresponding execution feedback queue through the driving service; and update the global status information of the current scanning task according to a preset status update frequency, and write the latest global status information into the corresponding status cache through the driving service; and after each grid cell scan is completed and the corresponding electron microscope image is generated, write the latest electron microscope image into the corresponding image cache through the driving service.

2. The natural language-driven SEM scanning task processing system according to claim 1, characterized in that, The prediction parameter set includes at least the target region parameters, the single-frame field of view size, the magnification range, the overlap rate, the acceleration voltage, the focus type, and the detector type. The target region parameters are composed of the coordinates of the four vertices of the target region; the single-frame field of view size is composed of the corresponding field of view width and field of view height; the focus type includes autofocus and manual focus; the detector type includes SE type, BSE type, and EDS type. The scan grid table consists of multiple scan grid records; each scan grid record consists of the single-frame field of view size and the coordinates of the grid center point. The task step flow is composed of multiple single-step operation nodes arranged sequentially; each single-step operation node includes a node identifier, a node type, and a node instruction set; the node type includes single-instruction nodes and multi-instruction nodes; the node instruction set consists of one or more first instructions; the first instruction includes an instruction code and instruction parameters; when the node type is a single-instruction node, the total number of instructions in the node instruction set is 1; when the node type is a multi-instruction node, the total number of instructions in the node instruction set is greater than 1. The scanning task planning model is based on a large language model or a multimodal large model, including at least the GPT series model, the DeepSeek series model, the SenseNova-MARS series model, the Wenxin series model, and the Tongyi Qianwen Qwen series model. The key paragraphs of the prompt instruction template include at least five parts: model role description paragraph, historical dialogue paragraph, user requirement paragraph, inference chain requirement paragraph, and output format constraint paragraph; wherein, the model role description paragraph, the inference chain requirement paragraph, the output format constraint paragraph, and the step instruction constraint paragraph are each a fixed natural language text; the historical dialogue paragraph and the user requirement paragraph are configurable paragraphs of the template, the historical dialogue paragraph is configured based on the historical context input to the template, and the user requirement paragraph is configured based on the planning instruction text input to the template; The model role description text is used to set the task role of the scanning task planning model as an SEM intelligent control assistant; and to describe the role task corresponding to the model: using the historical dialogue text as a context reference, and according to the thought chain reasoning steps set by the reasoning chain requirement text, multi-step deduction is performed according to the user requirement text, and atomic operation sequence parsing is performed on the electron microscope equipment operation steps during the deduction process, and the task step flow in the format specified by the output format constraint text is generated based on the parsing sequence. The reasoning chain requirement section is used to describe step-by-step reasoning steps of the thought chain of the scanning task planning model; the output format constraint section is used to describe the formatted text format of the task step flow output by the scanning task planning model, and the formatted text format includes at least TypeScript format and JSON format. The high-magnification sub-image sequence consists of multiple high-magnification sub-images; each high-magnification sub-image in the high-magnification sub-image sequence corresponds one-to-one with the scanning grid record in the scanning grid table, and each high-magnification sub-image is a grid electron microscope scan of a corresponding scanning grid in the target region corresponding to the target region parameter; when the low-magnification panoramic image is not empty, it is an electron microscope scan of the entire target region corresponding to the target region parameter; the scanning magnification of the high-magnification sub-images is greater than that of the low-magnification panoramic image; The Uni-AIMS analysis tool is used to perform scale identification, particle and pore target segmentation, particle size identification, pore diameter identification, particle distribution analysis, and pore distribution analysis on the electron microscope image to be analyzed. The analysis results of the electron microscope image output by the Uni-AIMS analysis tool include at least scale identification information, target segmentation mask image, statistical table of particle size / pore diameter data, and analysis chart of particle / pore distribution. The task interaction page adopts the layout style of an integrated development environment; the history dialogue area is used to display the human-computer interaction text content of the current scanning task line by line; the status panel is used to display the global status information of the current scanning task; the global status information includes at least scanning progress, vacuum status, electron beam parameters, detector status, and abnormal alarm information; the electron microscope window is used to load electron microscope images, and supports loading by selecting components or manually dragging and dropping, and supports selecting ROI areas on the electron microscope images loaded in the window by manually drawing. The execution instruction queue is used to sequentially store all the single-step operation nodes of the task step flow, with each queue message corresponding to one single-step operation node; the execution instruction queue has a configurable message read pointer for locating the current execution node; The execution feedback queue is used to store the instruction execution feedback corresponding to each single-step operation node; the instruction execution feedback includes the node identifier, the node type, and the instruction feedback set; the instruction feedback set consists of one or more first instruction feedbacks; the first instruction feedback includes the first instruction and the first feedback; when the node type is a single instruction node, the total number of instruction feedbacks in the instruction feedback set is 1; when the node type is a multi-instruction node, the total number of instruction feedbacks in the instruction feedback set is greater than 1. The state cache is used to store the global state information; The image buffer is used to store electron microscope images.

3. The natural language-driven SEM scanning task processing system according to claim 2, characterized in that, The parameter recognition model includes a text encoder, an image encoding module, an image encoder, an object detection model, a multimodal feature joint encoding module, and a parameter prediction module; The first model input terminal of the parameter recognition model is used to receive the scan overview text, the second model input terminal is used to receive the reference electron microscope image, and the model output terminal is used to output the predicted parameter set. The text encoder's input is connected to the first model's input, and its output is connected to the first input of the multimodal feature joint encoding module; the image encoding module's input is connected to the second model's input, and its output is connected to the second input of the multimodal feature joint encoding module; the image encoding module is also connected to both the image encoder and the target detection model; the multimodal feature joint encoding module's output is connected to the parameter prediction module's input; and the parameter prediction module's output is connected to the model's output. The text encoder is used to extract text features from the scanned overview text to obtain the corresponding feature vector X1, which is then sent to the multimodal feature joint encoding module. The feature vector X1 has a shape of 1×D1, where D1 is a preset first feature dimension. The image encoding module is used to identify whether the reference electron microscope image is empty. If the reference electron microscope image is empty, the corresponding feature vectors X2 and X3 are set to all zero vectors. If the reference electron microscope image is not empty, the image encoder is called to perform global feature extraction on the reference electron microscope image to obtain the corresponding feature vector X2. Using the ROI region bounding box as the detection target, the target detection model is called to perform target detection on the reference electron microscope image to obtain a set of corresponding target detection boxes. The target detection box with the highest confidence in the target detection box set is taken as the first matching box, and the first matching box is... The system identifies whether the bounding box is empty. If it is, the corresponding feature vector X3 is set to an all-zero vector; otherwise, the feature vector X3 is composed of the center point coordinates, width, and height of the first matching bounding box. The obtained feature vectors X2 and X3 are then sent to the multimodal feature joint encoding module. The shape of feature vector X2 is 1×D2, and the shape of feature vector X3 is 1×D3, where D2 and D3 are the preset second and third feature dimensions, respectively, and D3=4. The image encoder is implemented based on a residual network model, and the target detection model is implemented based on the YOLO series models. The multimodal feature joint encoding module is used to map the feature vectors X1, X2, and X3 to a preset target feature space using three linear layers, denoted as feature vectors Y1, Y2, and Y3 respectively; and to concatenate the feature vectors Y1, Y2, and Y3 to obtain the corresponding feature vector Y4; and to use an MLP model to encode the feature vector Y4 to obtain the corresponding feature vector Y5, which is then sent to the parameter prediction module. The mapping methods for feature vectors Y1, Y2, and Y3, and the encoding method for feature vector Y5 are as follows: ， ， ； ； W1 and b1 are the weight matrix and bias vector of the first linear layer. The weight matrix W1 has a shape of D4×D1, and the bias vector b1 has a shape of 1×D4, where D4 is the feature dimension of the target feature space. W2 and b2 are the weight matrix and bias vector of the second linear layer. The weight matrix W2 has a shape of D4×D2, and the bias vector b2 has a shape of 1×D4. W3 and b3 are the weight matrix and bias vector of the third linear layer. The weight matrix W3 has a shape of D4×D3, and the bias vector b3 has a shape of 1×D3. The feature vectors Y1, Y2, and Y3 all have a shape of 1×D4. The feature vector Y4 has a shape of 1×3D4. MLP1() is the inference function of the first MLP model. The parameter prediction module is used to identify whether the first matching box is empty. If it is, the corresponding target region parameter is set to the default region parameter of the SEM device corresponding to the current scanning task. Otherwise, the coordinates of the four vertices are calculated based on the center point coordinates, width, and height of the first matching box, and the four coordinates are used to form the corresponding target region parameter. The corresponding linear regression prediction head is used to predict the single-frame field of view size, the magnification range, the overlap rate, and the acceleration voltage. The corresponding classification prediction head is used to predict the focus type and the detector type. The corresponding prediction parameter set is formed by the obtained target region parameter, single-frame field of view size, magnification range, overlap rate, acceleration voltage, focus type, and detector type and then output.

4. The natural language-driven SEM scanning task processing system according to claim 2, characterized in that, The overlap rate optimization model includes a first feature extraction network, a second feature extraction network, a feature fusion layer, a feature encoding layer, an overlap rate prediction layer, and a prediction constraint module. The first model input terminal of the overlap rate optimization model is used to receive the ROI region map, the second model input terminal is used to receive the ROI region parameters, the single frame field of view size and the overlap rate, and the model output terminal is used to output the optimized overlap rate. The input of the first feature extraction network is connected to the input of the first model, and its output is connected to the first input of the feature fusion layer; the input of the second feature extraction network is connected to the input of the second model, and its output is connected to the second input of the feature fusion layer; the output of the feature fusion layer is connected to the input of the feature encoding layer; the output of the feature encoding layer is connected to the input of the overlap rate prediction layer; the output of the overlap rate prediction layer is connected to the input of the prediction constraint module; and the output of the prediction constraint module is connected to the model output. The first feature extraction network is implemented based on a residual network model; the first feature extraction network is used to perform global feature extraction on the ROI region map to generate the corresponding feature vector X. 1 Send to the feature fusion layer; The second feature extraction network is implemented based on an MLP model; the second feature extraction network is used to form an initial vector composed of the ROI region parameters, the single-frame field of view size, and the overlap rate; and to extract features from the initial vector to generate a corresponding feature vector X. 2 Send to the feature fusion layer; The feature fusion layer is used to perform vector concatenation on the feature vector X. 1 and the feature vector X 2 Vector concatenation yields the corresponding feature vector X. 3 Send to the feature coding layer; The feature encoding layer is implemented based on an MLP model; the feature encoding layer is used to process the feature vector X. 3 Feature encoding is performed to obtain the corresponding feature vector X. 4 Send to the overlap rate prediction layer; The overlap rate prediction layer is implemented based on a linear regression prediction network; the overlap rate prediction layer is used to predict the overlap rate based on the feature vector X. 4 The corresponding predicted overlap rate ρ is obtained by performing overlap rate prediction. * Send to the prediction constraint module; The prediction constraint module records the overlap rate of the model input as ρ. old And based on the overlap rate ρ old And a preset percentage α relative to the predicted overlap rate ρ * The corresponding optimized overlap rate ρ is obtained by applying constraints. new : ； Where 0 < α < 1.

5. The natural language-driven SEM scanning task processing system according to claim 2, characterized in that, The grid planning engine is specifically used when obtaining the corresponding scan grid table based on the optimized overlap rate scan grid planning: The center point coordinates of each scan grid on each grid row of the ROI region are calculated based on the ROI region parameters, the single-frame field of view size, and the optimized overlap rate to obtain the corresponding grid center point coordinates; and a corresponding scan grid record is formed by each grid center point coordinate and the single-frame field of view size; and all the scan grid records are sorted according to the scanning order to form the corresponding scan grid table.

6. The natural language-driven SEM scanning task processing system according to claim 2, characterized in that, The task planning agent is specifically used when performing compliance checks on the task step flow: Perform a sequential traversal of all the single-step operation nodes in the task step flow; and during this traversal, take the currently traversed single-step operation node as the current node; and poll each of the first instructions of the current node. During this round of polling, the first instruction currently being polled is taken as the current instruction; and the instruction code and instruction parameters of the current instruction are verified for compliance based on a preset instruction rule set; if the current instruction is successfully verified, the process moves to the next first instruction and continues polling until all first instructions of the current node have been polled; if the current instruction fails to be verified, the current round of polling and traversal ends, and the corresponding step flow verification result is set to failure; after all verifications in this round are successful, the process moves to the next single-step operation node and continues traversal until all single-step operation nodes have been traversed; and when all instructions of all single-step operation nodes are successfully verified, the corresponding step flow verification result is set to success; wherein, the instruction rule set includes multiple instruction rules; each instruction rule corresponds one-to-one with the instruction code; each instruction rule is used to specify the instruction code encoding of its corresponding instruction, and to specify the name, data type, parameter value range constraints of each instruction parameter of its corresponding instruction, and to specify whether its corresponding instruction has preconditions, and when its corresponding instruction has preconditions, to specify the logical judgment rules corresponding to the preconditions; The system then identifies whether the obtained step flow verification result is successful; if yes, the compliance check is confirmed to have passed; otherwise, the compliance check is confirmed to have failed.

7. The natural language-driven SEM scanning task processing system according to claim 2, characterized in that, The image stitching engine is specifically used when, based on the target region parameters carried in the current request, the scan grid table, the high-magnification sub-image sequence, and the low-magnification panoramic image, sub-image correction and full-sequence sub-image stitching are performed to obtain the corresponding high-fidelity panoramic image, which is then sent back to the current requester: Step 71: When the low-magnification panoramic image is not empty, based on the target region parameters and the scanning grid table, the corresponding low-magnification sub-images of each high-magnification sub-image sequence in the low-magnification panoramic image are confirmed, and distortion correction is performed on the corresponding high-magnification sub-images based on the feature point matching relationship, specifically: Each of the high magnification sub-images in the high magnification sub-image sequence is taken as the current high magnification sub-image; the magnification corresponding to the current high magnification sub-image is taken as the current high magnification; and the magnification corresponding to the low magnification panoramic image is taken as the current low magnification. The scanning grid record corresponding to the current sub-image in the scanning grid table is taken as the current record. Based on the target area parameters, the single-frame field of view size of the current record, and the grid center point coordinates, the scanning grid area corresponding to the current record on the low magnification panoramic image is confirmed, and the sub-image of the current scanning grid area is extracted as the current low magnification sub-image. The system performs feature point detection on the current high-magnification sub-image and the current low-magnification sub-image respectively according to a preset feature point detection algorithm to obtain the corresponding high-magnification sub-image feature point set and low-magnification sub-image feature point set; and confirms all matching feature point pairs of the high-magnification sub-image feature point set and the low-magnification sub-image feature point set based on a preset feature point matching algorithm; and identifies the projection transformation relationship between the current high-magnification sub-image and the current low-magnification sub-image based on all the obtained matching feature point pairs; and performs pixel coordinate transformation on the current high-magnification sub-image based on the identified projection transformation relationship; wherein, the feature point detection algorithm includes at least the SIFT algorithm, SURF algorithm, and ORB algorithm; and the feature point matching algorithm includes at least the Brute-Force algorithm and FLANN algorithm; Step 72: Construct a scanning grid plane space, and identify all adjacent sub-image pairs of the high-magnification sub-image sequence in the scanning grid plane space to obtain the corresponding set of adjacent sub-image pairs, specifically: The scanning grid plane space is constructed based on the target region parameters, the known single-frame field of view size, and the overlap rate; and all spatially adjacent sub-images of each high-magnification sub-image in the high-magnification sub-image sequence are retrieved based on the scanning grid plane space; and each high-magnification sub-image and its corresponding adjacent sub-image form a corresponding adjacent sub-image pair; all the obtained adjacent sub-image pairs are deduplicated; and all the deduplicated adjacent sub-image pairs form a corresponding adjacent sub-image pair set; The scanning grid plane space includes multiple scanning grids, the height and width of each scanning grid satisfying the single-frame field of view size; the overlap ratio of any two adjacent scanning grids in the plane space satisfies the overlap rate; each scanning grid corresponds to a high magnification sub-image; all spatially adjacent sub-images include the left neighbor sub-image, right neighbor sub-image, upper neighbor sub-image, and lower neighbor sub-image of the current sub-image; each of the adjacent sub-image pairs in the set of adjacent sub-image pairs is denoted as the corresponding sub-image pair Z. i 1 ≤ index i ≤ N, where N is the total number of subgraph pairs in the current subgraph pair set; each of the subgraph pairs Z i The two high-magnification sub-graphs are denoted as the corresponding sub-graph P. i,1 P i,2 ; Step 73, calculate the Z values ​​for each of the subgraph pairs based on the phase correlation method. i The corresponding relative displacement s i And by obtaining all the said relative displacements s i The corresponding set of relative displacements is as follows: Each of the subgraphs is paired with Z. i As the current subgraph pair; and for the subgraph P of the current subgraph pair i,1 P i,2 Performing a Fourier transform yields the corresponding spectrum F. i,1 Spectrum F i,2 ; and based on the spectrum F i,1 and the spectrum F i,2 F i,2 Calculate the corresponding cross-power spectrum. , For the spectrum F i,2 The complex conjugate of ε is a preset small constant to prevent the denominator from being zero; and the inverse Fourier transform of the cross-power spectrum R is performed to obtain the corresponding correlation function r(x,y), where x and y are the sub-graphs P. i,1 P i,2 The lateral and longitudinal relative displacements; and the extreme points (x, y) of the related function r(x, y). peak ,y peak Solve the problem; and calculate the lateral relative displacement x of the extreme points obtained from the solution. peak Longitudinal relative displacement y peak Let x be the corresponding lateral relative displacement. i Longitudinal relative displacement y i ; and by the lateral relative displacement x i and the longitudinal relative displacement y i The corresponding relative displacements s i ; and from all the said relative displacements s obtained i Form the corresponding set of relative displacements; Step 74, pair each of the subgraphs with Z i Subgraph P i,1 P i,2 The corresponding coordinates of the grid center point are marked as the corresponding subgraph center point c. i,1 c i,2 ; and based on all the said subgraphs, Z i The corresponding subgraph center point c i,1 c i,2 and the relative displacement s i Construct a translation optimization objective function E1; and based on a preset first optimization algorithm, determine the subgraph center point offset Δc that minimizes the translation optimization objective function E1. i,1 , △c i,2 An estimation is performed, and the coordinates of the grid center points of each of the high magnification sub-images are reset based on the estimation results; The translation optimization objective function E1 is: ； The first optimization algorithm includes the least squares method and the simulated annealing algorithm; During the optimization process, the sub-image center point offset Δc corresponding to the same high magnification sub-image is... i,1 or △c i,2 Maintaining global uniqueness; after optimization, because the center point offset △c of the sub-image corresponding to the same high magnification sub-image i,1 or △c i,2 To maintain global uniqueness, the offset of the center point of each globally unique sub-image corresponding to each high magnification sub-image is recorded as △c. Thus, the coordinates of the center point of the grid after reset for each high magnification sub-image are equal to the coordinates of the center point of the grid before reset plus △c. Step 75, perform the feature point detection algorithm on each of the sub-graph pairs Z. i Feature point detection is performed on the two subgraphs to obtain the corresponding first point set and second point set; and based on the feature point matching algorithm, all matching point pairs in the first point set and the second point set are confirmed to obtain the corresponding matching point pair set D. i ; Wherein, the set of matching point pairs D i With the subgraph pair Z i One-to-one correspondence; the set of matching point pairs D i Includes multiple matching point pairs d i,j 1 ≤ index j ≤ M, where M is the total number of matching point pairs in the current matching point pair set; each of the matching point pairs d i,j From the coordinates of point p i,j,1 Point coordinates p i,j,2 Composition; the point coordinates p i,j,1 For the corresponding subgraph P i,1 The coordinates of the point, the coordinates of the point p i,j,2 For the corresponding subgraph P i,2 The coordinates of the point; Step 76: Initialize a corresponding linear transformation function for each of the high magnification sub-images in the high magnification sub-image sequence as the initialization projection transformation relationship T. u 1 ≤ index u ≤ K, where K is the total number of subgraphs in the current subgraph sequence; and based on all the aforementioned projection transformation relationships T u and the set D of all the said matching point pairs i Construct a nonlinear distortion optimization objective function E2; and based on a preset second optimization algorithm, optimize all projection transformation relationships T that minimize the nonlinear distortion optimization objective function E2. u Make an estimate; Wherein, each of the projection transformation relationships T u Used to perform coordinate projection of the scanning grid plane space onto the corresponding high magnification sub-image; The nonlinear distortion optimization objective function E2 is: ； Let p be the coordinates of the point. i,j,1 The corresponding subgraph P i,1 The corresponding projection transformation relationship T u , Let p be the coordinates of the point. i,j,1 Through projection transformation relationship The coordinates of the projection point obtained after calculation; Let p be the coordinates of the point. i,j,2 The corresponding subgraph P i,2 The corresponding projection transformation relationship T u , Let p be the coordinates of the point. i,j,2 Through projection transformation relationship The coordinates of the projection point obtained after calculation; The second optimization algorithm includes the Levenberg-Marquardt algorithm, the Gauss-Newton algorithm, and the simulated annealing algorithm; Step 77, based on each estimated projection transformation relationship T u The pixel coordinates of the corresponding high magnification sub-image are transformed by the pixel coordinates in the scanning grid plane space to obtain the corresponding grid plane sub-image; and the obtained grid plane sub-images are stitched together in spatial arrangement to obtain the corresponding high-fidelity panoramic image, which is then sent back to the current requester.

8. The natural language-driven SEM scanning task processing system according to claim 2, characterized in that, The interactive terminal is specifically used when the corresponding task step flow is obtained by performing step flow generation processing based on the submission information entered by the user on the current task interaction page, the parameter recognition agent, the grid planning engine, and the task planning agent: When a user submits a piece of natural language text through the dialog input box on the current task interaction page, the current natural language text is recorded as the first text; the electron microscope image loaded in the electron microscope window on the current task interaction page is used as the current electron microscope image; and the first text is added to the history dialog area. The first text is used as the corresponding scan overview text; the current electron microscope image is checked to see if it is empty. If it is, the reference electron microscope image is set to empty; otherwise, the current electron microscope image is checked to see if it has a Region of Interest (ROI) bounding box. If it is, the electron microscope image with the ROI bounding box is used as the reference electron microscope image; otherwise, the current electron microscope image is used as the reference electron microscope image. The parameter recognition request carrying the scan overview text and the reference electron microscope image is sent to the parameter recognition agent. The scanning parameter configuration sent back by the parameter recognition agent is displayed to the current user via a pop-up window. Each parameter in the scanning parameter configuration is provided with a modification interface on the pop-up window. After the user clicks the confirmation button on the pop-up window, the scanning parameter configuration is updated based on the latest parameter information on the pop-up window, and the pop-up window is closed. The prediction parameter set includes at least the target area parameters, the single-frame field of view size, the magnification range, the overlap rate, the acceleration voltage, the focus type, and the detector type. The current electron microscope image is identified. If the current electron microscope image is empty, the corresponding ROI region image is set to empty. If the current electron microscope image is not empty and does not have the ROI region marker frame, the corresponding ROI region image is set as the current electron microscope image. If the current electron microscope image is not empty and has the ROI region marker frame, the marker frame sub-image on the current electron microscope image corresponding to the ROI region marker frame is used as the corresponding ROI region image. The corresponding ROI region parameters are set to the target region parameters configured in the scanning parameters. The corresponding single-frame field of view size and overlap rate are extracted from the scanning parameter configuration; and the grid planning request carrying the ROI region map, the ROI region parameters, the single-frame field of view size and the overlap rate is sent to the grid planning engine. It also receives the optimized overlap rate and the scanned grid table sent back by the grid planning engine; The overlap rate in the scan parameter configuration is updated based on the optimized overlap rate; and the scan parameter configuration and the scan grid table are added to the history dialog area. And the first text is used as the corresponding planning instruction text; The entire content of the historical dialogue area is used as the corresponding historical context; the task planning request carrying the planning instruction text and the historical context is sent to the task planning agent; and the task step flow sent back by the task planning agent is sent back.

9. The natural language-driven SEM scanning task processing system according to claim 2, characterized in that, Specifically, the interactive terminal is used when the image stitching request is generated based on the cached image of the cache service and sent to the image stitching engine: The electron microscope image loaded in the electron microscope window of the current task interaction page is used as the current electron microscope image; and the current electron microscope image is identified. If the current electron microscope image is empty, the corresponding low magnification panoramic image is set to empty. If the current electron microscope image is not empty and does not have a ROI region marker box, the corresponding low magnification panoramic image is set as the current electron microscope image. If the current electron microscope image is not empty and has the ROI region marker box, the marker box sub-image on the current electron microscope image corresponding to the ROI region marker box is used as the corresponding low magnification panoramic image. The electron microscope images cached in the image cache area corresponding to the current scanning task on the cache service are taken as the corresponding high magnification sub-images, and all the obtained high magnification sub-images are sorted in chronological order to form the corresponding high magnification sub-image sequence; and after obtaining the high magnification sub-image sequence, all cached images in the current image cache area are cleared; The image stitching request, which carries the target area parameters, the scan grid table, the high magnification sub-image sequence, and the low magnification panoramic image, is sent to the image stitching engine.

10. The natural language-driven SEM scanning task processing system according to claim 2, characterized in that, The interactive terminal is specifically used when the current task interactive page is updated based on the electron microscopy image analysis results sent back by the analytical agent: The charts in the electron microscopy analysis results are displayed via pop-up windows; the scale identification information, various data statistics tables, and various analysis tables in the electron microscopy analysis results are converted into text information and added to the history dialogue area of ​​the current task interaction page; the foreground pixels of the target segmentation mask image in the electron microscopy analysis results are colored, and the resulting colored mask image is superimposed on the electron microscopy image to be analyzed to obtain the corresponding colored electron microscopy image, which is then loaded into the electron microscopy window of the previous task interaction page.

11. The natural language-driven SEM scanning task processing system according to claim 2, characterized in that, The scheduling service is specifically used when the SEM device corresponding to the device identifier is invoked through the driver service to execute the current task step flow step by step and when the end of the current task is sent to the interactive terminal: By querying the locally pre-set device parameter library through the device identifier, the device parameter set of the SEM device corresponding to the device identifier is obtained; wherein, the device parameter library includes multiple device parameter sets, and each device parameter set corresponds to the device identifier of the SEM device; the device parameter set of each SEM device is used to define the instruction code encoding of all device instructions of the current device, define the name, data type, parameter value range constraints of all device instruction parameters of the current device, define whether each device instruction of the current device has prerequisites, and define the prerequisite condition logic judgment rules for device instructions with prerequisites; Then, all the single-step operation nodes of the current task step flow are pushed into the corresponding execution instruction queue in sequence; and the message reading pointer is initially positioned at the first single-step operation node. Each time the message read pointer is positioned, the single-step operation node pointed to by the message read pointer is taken as the current node; the node instruction set of the current node is taken as the current instruction set; and the execution conditions of all instructions in the current instruction set are checked according to the global state information cached in the state cache area corresponding to the current scanning task on the cache service and the current device parameter set. If the check fails, the abnormal alarm information of the global state information cached in the state cache area is set according to the reason for failure, the current task is stopped, and the termination notification specifically set to abnormal termination is sent to the interactive terminal. If the check passes, the current instruction set is forwarded to the SEM device corresponding to the device identifier by calling the standardized interface of the driver service; and a receiving wait is performed after the instruction is forwarded; after the waiting time exceeds the preset standard waiting time, the latest instruction execution feedback added in the execution feedback queue is identified as matching the current node; if they do not match, a preset timeout is set. The operation action is identified. If the timeout operation action is "stop", the current task is stopped and a termination notification specifically set to "abnormal termination" is sent to the interactive terminal. If the timeout operation action is "retransmission", the current instruction set is retransmitted to the current SEM device, and after retransmission, the next round of waiting begins. If a match is found, the system identifies whether there is abnormal feedback in the current instruction execution feedback. If abnormal feedback exists, the system identifies the operation action corresponding to the current abnormal feedback based on a preset abnormal feedback-operation action correspondence rule. If the current operation action is "stop", the current task is stopped and a termination notification specifically set to "abnormal termination" is sent to the interactive terminal. If the current operation action is "rollback", the message reading pointer is positioned to the previous single-step operation node. If there is no abnormal feedback, the system identifies whether the single-step operation node corresponding to the current message reading pointer is the last node. If so, the current task is stopped and a termination notification specifically set to "successful termination" is sent to the interactive terminal. Otherwise, the message reading pointer is positioned to the next single-step operation node.

Images