Method and device for controlling imaging equipment

Abstract

The invention discloses a method and device for controlling imaging equipment, and belongs to the technical field of automatic control. The method comprises the following steps: acquiring an operation instruction for the imaging equipment, wherein the operation instruction is used for indicating the imaging equipment to execute a target operation; through a first conversion module, the operation instruction is converted into operation information for the imaging equipment, and the operation information is used for determining target operation indicated to be executed by the imaging equipment; and through a second conversion module, according to an operation document of the imaging equipment, converting the operation information into a control program for the imaging equipment for controlling the imaging equipment. According to the method disclosed by the invention, the imaging equipment can automatically execute the operation according to the operation instruction of the user, so that the time cost of learning and mastering a complex equipment operation method by the user is saved, and the working efficiency of the user is improved.

Description

Technical Field

[0001] The present application generally relates to the field of automation control technology. More specifically, the present application relates to a method and apparatus for controlling an imaging device. Background Art

[0002] Scanning Electron Microscope (SEM) is a high-resolution imaging device widely used in fields such as materials science, life science, nanotechnology, and semiconductor industry. However, the process of using SEM to observe samples involves a large number of parameter settings and adjustment operations, which causes considerable difficulties for users. In the prior art, although some SEM operations have been simplified through automatic scripts and graphical user interfaces (GUIs), there are still a large number of operations that require users to master independently, and it is impossible to achieve full automation of SEM operations.

[0003] In view of this, there is an urgent need to provide a method for controlling an imaging device so that the SEM can automatically perform operations according to user operation instructions, thereby saving the user's time cost of learning and mastering complex SEM operation methods and improving the user's work efficiency. Summary of the invention

[0004] In order to at least solve one or more technical problems described in the above background technology section, the present application proposes the following technical solutions and multiple embodiments thereof.

[0005] In a first aspect, the present application discloses a method for controlling an imaging device, comprising: obtaining an operation instruction for the imaging device, the operation instruction being used to instruct the imaging device to perform a target operation; converting the operation instruction into operation information for the imaging device through a first conversion module, for determining the target operation that the imaging device is instructed to perform; and converting the operation information into a control program for the imaging device according to an operation document of the imaging device through a second conversion module, for controlling the imaging device.

[0006] In a second aspect, the present application discloses a device, comprising a processor configured to execute program instructions; and a memory configured to store the program instructions, so that when the program instructions are loaded and executed by the processor, the device executes the method according to the first aspect.

[0007] By utilizing the method and apparatus for controlling an imaging device proposed in the present application, the imaging device can automatically perform a target operation according to a user's operating instructions, thereby improving the usability of the imaging device, reducing the difficulty for users to use the device, and improving work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become easy to understand. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein: Figure 1 An exemplary flow chart of a method for controlling an imaging device in some embodiments of the present application is shown.

[0009] Figure 2 An exemplary schematic diagram shows the conversion relationship between operating instructions, operating information and control programs in some embodiments of the present application.

[0010] Figure 3 An exemplary flow chart of a method for obtaining operating instructions for an imaging device in some embodiments of the present application is shown.

[0011] Figure 4 An exemplary flow chart of a method for converting operation information into a control program for an imaging device in some embodiments of the present application is shown.

[0012] Figure 5 An exemplary flowchart of the first sub-operation process in some embodiments of the present application is shown.

[0013] Figure 6 An exemplary flow chart of a method for controlling an imaging device to acquire a first image based on adjustment parameters and adjustment positions in some embodiments of the present application is shown.

[0014] Figure 7 A schematic diagram of the method architecture in one embodiment of the present application is shown.

[0015] Figure 8 A block diagram showing a hardware configuration of an apparatus that can implement an embodiment of the present application is shown. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0017] It should be understood that the terms "include" and "comprising" used in the specification and claims of the present application indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0018] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the present specification and claims, the singular forms of "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" used in the present specification and claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0019] As used in this specification and claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0020] The specific implementation of the present application is described in detail below with reference to the accompanying drawings.

[0021] SEM is a high-resolution imaging device widely used in fields such as materials science, life science, nanotechnology, and the semiconductor industry. SEM uses a focused electron beam to scan the surface of a sample. The electron beam interacts with the sample atoms to produce a variety of collectible signals, such as backscattered electrons, secondary electrons, characteristic X-rays, cathode luminescence, sample current, etc. By collecting signals including but not limited to the above, information characterizing the surface morphology and composition of the sample can be obtained, thereby generating a high-magnification image of the sample surface. After preparing the sample, the sample is mounted on the sample stage. Subsequently, the use of SEM to observe the sample usually includes the following operations: Check the device connection status, including checking the device power connection and checking whether the communication between devices is normal.

[0022] Adjust SEM intrinsic parameters, including magnification, contrast, brightness, focus, signal-to-noise ratio, accelerating voltage, electron beam current, etc.

[0023] Adjust the position of the sample stage and find the area of ​​interest in the sample. The sample stage usually has a support surface, which can be any shape to accommodate different types of samples. The sample stage can be moved and rotated to observe the sample at different angles and positions. Some sample stages are also equipped with a tilting device to tilt the sample at a certain angle to better observe the side of the sample or the area of ​​interest.

[0024] Capture and analyze the captured images. After capturing the image, you can observe the sample features in the image, measure the size of the sample, perform quantitative analysis as needed, and use relevant software and algorithms for image analysis.

[0025] In addition, since X-rays are released during the operation of SEM, which are potentially harmful to the human body, some safety precautions need to be taken when operating the instrument. Therefore, in the process of using SEM to observe samples, a large number of parameter settings and adjustment operations are involved. The operation of traditional SEM equipment is often manually adjusted through the front panel of the equipment or the computer software interface. Finding the area of ​​interest in the sample also requires manual operation, and then importing other analysis software to complete the image analysis. However, for researchers in a specific scientific research field, learning and mastering the professional knowledge in their field has taken up a lot of energy. They are relatively unfamiliar with the complex internal structure, imaging principles, and operation and control methods of SEM, and it is difficult for them to complete complex operation tasks.

[0026] In response to the problem that SEM is difficult to operate, some modern SEM devices support the automation of routine tasks through scripts, but these scripts usually require users to have certain programming knowledge, and the script writing and debugging process is relatively complicated; some SEM manufacturers provide GUIs, allowing users to set up devices and acquire images through GUIs, but the use of GUIs still requires users to be familiar with various parameters and operating procedures. In short, the existing technology can only achieve partial automation of SEM operations, but it is impossible to achieve that the equipment can fully automatically complete operating tasks according to instructions.

[0027] Nowadays, with all walks of life accelerating their digital transformation, artificial intelligence (AI) has gradually penetrated into all aspects of enterprise design, R&D, production, management, marketing, and services, and is a key technology that drives enterprises to upgrade to digital intelligence. As a rising star in the field of AI, large language models (LLM) represented by ChatGPT have greatly promoted the development of artificial intelligence generated content (AIGC), enabling machines to understand human intentions and achieve high-quality "human-computer dialogue". With its outstanding ability to answer questions, create content, and write code, LLM allows people to intuitively and truly experience the tremendous changes and efficiency improvements brought about by the advancement of artificial intelligence technology.

[0028] LLM provides a solution for realizing the automated operation of SEM. However, since there is no LLM tailored for SEM at present, and the general LLM cannot stably and reliably generate control solutions according to the user's operating instructions, how to realize the complete automation of SEM operation is still a technical problem faced by the industry. In view of this, in view of the problem that the prior art cannot realize the SEM to complete the operation task completely automatically according to the user's instructions, the present application provides a method for controlling an imaging device, so as to effectively utilize the logical reasoning ability and generation ability of a large language model, so as to realize the SEM to automatically perform the operation task according to the user's operating instructions, thereby saving the user's time cost of learning and mastering complex SEM operation methods and improving the work efficiency of operators.

[0029] Figure 1 An exemplary flow chart of a method for controlling an imaging device in some embodiments of the present application is shown. Figure 1 As shown, the method includes: step 101, obtaining an operation instruction for the imaging device, the operation instruction is used to instruct the imaging device to perform a target operation; step 102, converting the operation instruction into operation information for the imaging device through a first conversion module, which is used to determine the target operation that the imaging device is instructed to perform; step 103, converting the operation information into a control program for the imaging device according to the operation document of the imaging device through a second conversion module, which is used to control the imaging device.

[0030] First of all, it can be understood that the method disclosed in this application is not only applicable to SEM, but also to other imaging devices for capturing, recording or displaying images, including but not limited to X-ray machines, Computed Tomography (CT) scanners, Magnetic Resonance Imaging (MRI) devices, optical microscopes, cameras, video cameras, scanners, etc. Operating this type of imaging equipment often requires professional imaging knowledge and complex equipment operation methods. According to the method disclosed in this application, the imaging device can automatically perform the target operation according to the user's operating instructions, thereby improving the ease of use of the imaging device, reducing the difficulty of users using the device, and improving work efficiency.

[0031] For step 101, it can be understood that the operations that the imaging device can perform include but are not limited to checking the device connection status, adjusting the device inherent parameters, shooting and analyzing images. The user can instruct the imaging device to perform the target operation by inputting an operation instruction. In some embodiments, the method of obtaining the user's operation instruction includes but is not limited to: providing an operation interface with a text input function, the user inputs the operation instruction through a keyboard or handwriting in the operation interface, and the imaging device receives it; providing an operation interface with a voice input function, the user inputs the operation instruction through voice in the operation interface, and the imaging device receives it; for various operations supported by the imaging device, specific functional buttons are provided respectively, the user clicks the button to input the operation instruction, and the imaging device receives it. In these embodiments, the operation instruction refers to an instruction issued by the user through a certain behavior (inputting text, voice dialogue, clicking a button, etc.) to instruct the imaging device to perform an operation. By establishing a conversion relationship between the user behavior and the operation instruction, the user's behavior can be converted into a specific operation instruction.

[0032] For step 102, it can be understood that, for the imaging device, it is designed to have a control system implemented by software and a hardware device system including components such as an electron gun, an electromagnetic lens, and a scanning coil. Under the control of the software control system, each hardware device cooperates to perform operations. However, the imaging device is not designed to understand the operation instructions issued by the user through behavior; in addition, in the software control system of the imaging device, all executable operations and corresponding control instructions are strictly defined, and the operation instructions issued by the user through text or voice have a certain degree of arbitrariness. In other words, the operation instructions issued by the user cannot be directly corresponded to the control instructions in the control system, and will be accompanied by irrelevant information. Therefore, the present application proposes a first conversion module to convert the operation instructions issued by the user into operation information for the imaging device.

[0033] In some embodiments, the first conversion module includes a first large language model, such as the ChatGPT series large language models: ChatGPT-3, ChatGPT-3.5, ChatGPT-4; BERT, PaLM model, LLaMA model, Wenxinyiyan model. In other embodiments, the first large language model is guided to output by a prompt word (Prompt), for example, a prompt word can be set, "As an imaging device operator, you need to extract the operation instructions input by the user. If the operation instructions are not extracted, you need to prompt the operation instructions supported by the device" or "As an imaging device image shooting technician, you need to determine whether the user's shooting operation instructions are correct. If the user's shooting operation instructions are incorrect, you need to prompt the correct operation instructions" or "Please extract the user's input operation instructions for the imaging device and convert the operation instructions into operation information for the imaging device." For those skilled in the art, appropriate prompt words can be set for guidance according to the functions and output content of the large language model and other conditions, and this application does not limit the content of this prompt word.

[0034] Continuing to step 102, the first conversion module converts the operation instruction into operation information for the imaging device, and the operation information includes the target operation that the imaging device is instructed to perform. For example, after the first large language model converts the operation instruction input by the user into operation information, the operation information includes operations associated with the operation of the imaging device, such as "shoot", "move", and "set magnification".

[0035] As for step 103, it can be understood that the software control system controls the various hardware devices in the imaging device to perform operations in collaboration based on a written or dynamically generated control program, so it is necessary to convert the aforementioned operation information into a control program to control the imaging device to perform operations. In terms of content, the operation document of the imaging device includes a description of the functions and usage of the imaging device, which is used to provide a reference for the second conversion module to generate a control program. That is to say, the second conversion module generates a control program within the scope of the content contained in the operation document, thereby ensuring the correctness of the generated control program. Specifically, the second conversion module converts the relevant description in the operation document into a corresponding code library. After receiving the operation information, the second conversion module extracts the code snippets associated with the operations contained in the operation information in the code library, and then organizes the code snippets into a control program.

[0036] Continuing with step 103, in some embodiments, the second conversion module includes a second language model that can write a code program according to the given input information, such as the ChatGPT series language model, the Code LLaMA model, the GitHubCopilot model, etc. In other embodiments, the second language model is guided to output by prompt words, for example, the prompt words can be set to "first think about the meaning of the user operation instruction, disassemble the user operation instruction into an execution action according to the imaging device API interface, and execute the function call of the interface after confirmation" or "Please generate a control program for the user operation instruction according to the operation document of the imaging device."

[0037] It can be understood that according to the method for controlling the imaging device proposed in the present application, the user is allowed to operate the imaging device through certain actions (inputting text, voice conversation, etc.), thereby achieving a more intuitive and convenient interaction method, thereby reducing the operating difficulty of imaging devices including SEM and improving experimental efficiency.

[0038] To facilitate readers’ understanding, Figure 2 Taking the user's voice interaction as an example, the conversion between operation instructions, operation information and control programs in some embodiments of the present application is shown. In the illustrated example, the user inputs by voice: "I want to observe the microstructure on the surface of the sample, please set the acceleration voltage to 15kV, the electron beam current to 5nA, adjust the sample stage tilt angle to 30°, and then take a picture", and the imaging device uses the received voice signal as the operation instruction issued by the user; then, the first conversion module converts the operation instruction into operation information, wherein the operation information includes the target operation that the imaging device is instructed to perform, such as "set the acceleration voltage to 15kV", "take a picture"; then, the second conversion module converts the operation information into a control program, which is read and executed by the computing device in the imaging device, thereby controlling the action of the imaging device.

[0039] Figure 3 An exemplary flow chart of a method for obtaining an operation instruction for an imaging device in some embodiments of the present application is shown. It can be understood that: Figure 3 It's in front Figure 1 A specific implementation of step 101. Figure 3 As shown, in some embodiments, obtaining operation instructions for the imaging device includes: step 301, obtaining user behavior instructions; step 302, in response to extracting specific information associated with the imaging device from the behavior instructions, obtaining operation instructions for the imaging device from the behavior instructions.

[0040] For step 301, it can be understood that the user's behavior instruction refers to an instruction issued by the user through a certain behavior (input text, voice dialogue, click button, etc.), which is characterized by being close to the user's expression. In some embodiments, the behavior instruction includes natural language instructions and gesture instructions. The natural language instructions include but are not limited to instructions issued by voice, input text or symbols, and the gesture instruction refers to an instruction issued by the user through gestures, which can be received by the shooting device configured on the imaging device.

[0041] For step 302, it can be understood that, unlike the instructions within a specific range provided by the control system of the imaging device, the behavioral instructions input by the user have higher flexibility and a wider range; for example, when the user inputs by voice, some questions unrelated to the imaging device can be asked. In this case, it is necessary to screen the behavioral instructions input by the user so that only the behavioral instructions related to the operation of the imaging device are converted and processed, so as to reduce unnecessary operation of the large language model and improve the efficiency of the control method. In some embodiments, after the user's behavioral instructions are obtained, the behavioral instructions are extracted and processed using the first large language model, and in response to extracting specific information associated with the imaging device from the behavioral instructions, operation instructions for the imaging device are obtained from the behavioral instructions.

[0042] Continuing with step 302, in some embodiments, the specific information includes at least one of the following: sample information, operating parameters, and target objects. It is understandable that the sample information is the information of the sample object observed by the imaging device, including but not limited to the sample name, sample type, sample humidity, and other information; the operating parameters are the operation-related parameters specified when the imaging device performs an operation, such as when performing a shooting operation, it is necessary to specify parameters such as brightness, magnification, pixel size, etc.; and the target object refers to the object that is expected to be observed or photographed from the sample. In other embodiments, in order to ensure that the user's behavior instructions are correct and effective, the specific information must at least include sample information, operating parameters, and target objects, so as to avoid generating erroneous operation information and controlling the imaging device to perform erroneous operations.

[0043] In some embodiments, converting the operation instruction into operation information for the imaging device includes: converting the operation instruction into operation information for the imaging device according to the specific information.

[0044] As mentioned above, after obtaining the user's behavior instructions, the first large language model can be used to extract specific information associated with the imaging device from the behavior instructions, such as sample information, multiple operations, multiple operation parameters, and target objects. However, the specific information extracted is independent of each other and is not sufficient to guide the imaging device to perform operations accurately. For ease of understanding, here is an example: the same is to control the imaging device to shoot the sample at a magnification of 100 times. One possible user behavior instruction is "shoot the sample at a magnification of 100 times", and another possible user behavior instruction is "shoot the sample at a magnification of 100 times"; although in both cases, the large language model can extract the same specific information "shoot" and "magnification of 100 times", but in different orders to perform the operations corresponding to the two specific information, but will get different results; the first result is to shoot first, and then adjust the magnification to 100 times; the other result is to adjust the magnification to 100 times first, and then shoot. In this case, based on the logical reasoning and generation capabilities of the large language model, the extracted specific information can be organized to generate reasonable and detailed operation information.

[0045] Figure 4 An exemplary flow chart of a method for converting operation information into a control program for an imaging device in some embodiments of the present application is shown. It can be understood that Figure 4 It's in front Figure 1 A specific implementation of step 103. Figure 4 As shown, in some embodiments, converting the operation information into a control program for the imaging device includes: step 401, converting the operation information into an operation process for the imaging device, and the operation process is used to plan the action of the imaging device; step 402, converting the operation process into a control program for the imaging device.

[0046] It can be understood that the operation instructions input by the user are usually general or unclear. According to the method of the present application, the operation information converted according to the operation instructions is further processed to generate a detailed operation process, and then a control program is generated according to the detailed operation process, thereby achieving precise control of the imaging device.

[0047] In some embodiments, converting the operation information into an operation process for an imaging device includes: in response to the operation instruction including an image acquisition instruction for a target object, converting the operation information into an operation process including the following contents: a first sub-operation process for controlling the imaging device to acquire a first image; a second sub-operation process for locating a region of interest in the first image; and a third sub-operation process for controlling the imaging device to acquire a second image associated with the region of interest.

[0048] It can be understood that the embodiment of the present application first controls the imaging device to capture a first image of the sample as a whole through a first sub-operation process, then determines the position of the target object in the sample from the first image through a second sub-operation process, and finally controls the imaging device to capture a clear image of the target object through a third sub-operation process so as to accurately respond to the user's operating instructions.

[0049] It is understandable that when the user's operation instruction contains relevant descriptions of the target object and the shooting action, the operation information converted from the operation instruction will correspondingly contain information on shooting the target object. Therefore, based on the user's operation instruction, the first language model can be used to determine whether the operation instruction includes an image acquisition instruction for the target object; or, based on the operation information converted from the operation instruction, the second language model can be used to determine whether the operation instruction includes an image acquisition instruction for the target object.

[0050] Figure 5 An exemplary flow chart of the first sub-operation process in some embodiments of the present application is shown, such as Figure 5 As shown, in some embodiments, the first sub-operation process includes: step 501, adjusting the inherent parameters of the imaging device to obtain adjusted parameters; step 502, adjusting the position of the target object in the field of view of the imaging device to obtain an adjusted position; and step 503, controlling the imaging device to acquire a first image based on the adjusted parameters and the adjusted position.

[0051] In some embodiments, the inherent parameters are adjusted based on the properties of the sample, the user's observation purpose and the observation requirements. For those skilled in the art, appropriate parameters can be selected according to the specific situation, or the parameters can be optimized through multiple experiments, and this application does not limit this. Taking the acceleration voltage, an inherent parameter of SEM, as an example, a higher acceleration voltage should be selected for situations where high-resolution observation of surface fine structures is required; and a lower acceleration voltage should be selected for samples with poor conductivity to reduce the charging effect. In some embodiments, the inherent parameters include at least one of the following: magnification, brightness, contrast, signal-to-noise ratio, acceleration voltage, electron beam current and focusing parameters.

[0052] For step 502, it is understood that the field of view of the imaging device refers to the range of physical space that the device can observe. In some embodiments, adjusting the position of the target object in the field of view of the imaging device includes: adjusting the position of the target object and / or adjusting the field of view of the imaging device. It is understood that the position of the target object can be adjusted by moving the sample up and down, left and right, or raising and lowering the stage, and adjusting the tilt angle of the stage. For a specific imaging device, the field of view of the imaging device can also be adjusted by adjusting inherent parameters; for example, for SEM, the direction of collecting signals can be changed by adjusting the position and angle of the detector, thereby changing the range of the field of view.

[0053] After adjusting the intrinsic parameters and the position of the target object in the field of view of the imaging device, the imaging device can be controlled to obtain a first image of the target object. In some embodiments, in order to provide a sufficiently clear image of the target object that is convenient for the user to observe and analyze, the imaging device is controlled to obtain the first image based on the adjusted parameters and the adjusted position, including: based on the adjusted parameters and the adjusted position, the imaging device is controlled to obtain a sample image; until the quality of the sample image is qualified, the following steps are cyclically performed: updating the intrinsic parameters of the imaging device to obtain an updated parameter; updating the position of the target object in the field of view of the imaging device to obtain an updated position; and based on the updated parameter and the updated position, controlling the imaging device to obtain a sample image; using the sample image as the first image.

[0054] Figure 6 An exemplary flow chart of a method for controlling the imaging device to acquire a first image based on adjustment parameters and adjustment positions in some embodiments of the present application is shown. Figure 6 As shown, first enter step 601, based on the adjustment parameters and the adjustment position, control the imaging device to obtain a sample image; then enter step 602 to determine whether the quality of the sample image is qualified: if qualified, enter step 606 and use the sample image as the first image; otherwise, execute steps 603, 604, 605, and 602 cyclically until it is determined in step 602 that the quality of the sample image is qualified, wherein steps 603-605 are the above-mentioned updating of inherent parameters, the position of the target object in the field of view of the imaging device, and the acquisition of the sample image, which will not be repeated here. It should be pointed out here that the quality of the sample image can be judged based on the position of the target object in the sample image, the clarity, and whether it is convenient for the user to observe and analyze. This application does not limit this judgment method.

[0055] In some embodiments, the second sub-operation process includes: segmenting the target object from the first image to obtain a target area corresponding to the target object; and determining the target area as a region of interest.

[0056] It is understandable that the image segmentation algorithm is used to input an image and can output a segmented image of the same size as the input image, and each pixel in the output image is marked as belonging to a certain category. Then, the first image is input into a given image segmentation algorithm, and the pixels marked as belonging to the target object category are identified from the output image to obtain the target area corresponding to the target object. In some embodiments, considering the diversity of samples and target objects, the Segmentation Anything Model (SAM) is used as an image segmentation algorithm to segment the target object from the first image. SAM is trained on a large dataset containing millions of images and billions of masks. The model can migrate zero samples to new image distributions and tasks, thereby improving the accuracy of the segmentation results of the first image.

[0057] After the position of the target object is determined, the imaging device may be controlled to acquire an image of the target object. In some embodiments, the third sub-operation process includes: controlling the imaging device to acquire a second image with the region of interest as the center of the field of view.

[0058] In some embodiments, the second conversion module includes an artificial intelligence model; through the second conversion module, the operation information is converted into a control program for the imaging device according to the operation document of the imaging device, including: establishing a model knowledge base according to the operation document of the imaging device, and the model knowledge base is used to: guide the artificial intelligence model to convert the operation information into a control program for the imaging device.

[0059] It is understandable that the artificial intelligence model can be the second language model mentioned above. By establishing a model knowledge base for the large language model, the output of the large language model can be guided and standardized. In some embodiments, the model knowledge base includes a code base generated according to the operation document and an N-dimensional vector space obtained by mapping each content in the operation document. , where N is the length of the vector used to represent each content in the operation document. Specifically: each content in the operation document is a description of the function and usage of the device, and the second largest language model generates a corresponding code based on each content, which can be executed by the imaging device and perform corresponding operations; after receiving the operation information, the second largest language model also maps the operation information to the above-mentioned N-dimensional vector space to obtain a vector Q, and then extracts a vector group P in V with a high correlation or similarity with Q, and determines the code snippet in the code library corresponding to each vector in the vector group P, and then generates a control program based on the code snippet.

[0060] In some embodiments, the operation document includes: operations for the imaging device, and application programming interfaces corresponding to the operations. Specifically, the operation document consists of a user instruction document and an imaging device hardware protocol, wherein the contents of the user instruction document include, but are not limited to: an overview of the imaging device, operations supported by the imaging device and steps corresponding to the operations, imaging device troubleshooting methods, etc. The contents of the imaging device hardware protocol include, but are not limited to: the hardware interface of the imaging device, including the application programming interface (API) corresponding to the device operation, i.e., the function methods and usable classes and objects that need to be called when writing the control program; the communication method and protocol between the imaging device and external devices; the communication protocol between the hardware components inside the imaging device; the electrical connection method and interface standard between the hardware components of the imaging device, etc.

[0061] Exemplary Embodiments This section provides a complete embodiment of the above method according to the present application. The embodiment is a SEM control method based on a natural language model. By integrating advanced artificial intelligence technology, an adaptation layer is developed based on the underlying hardware interface of the SEM, so that the LLM can be compatible with the hardware interface of the SEM.

[0062] Figure 7 FIG. 2 shows a schematic diagram of the method architecture of this embodiment. Figure 7 As shown, the method architecture of this embodiment includes the hardware components of the scanning electron microscope and the intelligent agent service software library, and the intelligent agent service software library enables the hardware components of the SEM to perform operations through the SEM hardware control protocol. The hardware components of the SEM include high-voltage modules, sample stage modules, aperture modules, vacuum modules, detectors, deflection amplifiers, etc. The intelligent agent service software library includes the AutoGen framework, the SAM image segmentation model, the LLM, and a knowledge base consisting of the "Scanning Electron Microscope Instructions" and "Scanning Electron Microscope Hardware Protocol" documents. Among them, AutoGen is an open source programming framework for building AI agents and promoting cooperation between multiple agents to solve tasks.

[0063] The development work involved in implementing this embodiment includes: 1. Develop hardware protocols and API interfaces so that LLM can send instructions and receive device status information; 2. To ensure that SEM has the necessary hardware support to support the operation of LLM, such as high-speed data transmission interface and sufficient computing power, adapt the existing SEM hardware; 3. Integrate the LLM model, AutoGen framework and SAM image segmentation model into the SEM’s operating software to ensure seamless collaboration between new and old systems.

[0064] 4. Define two intelligent agents (Agents) based on the scanning electron microscope operation process: assistant agent and user agent.

[0065] The workflow of this embodiment includes the following steps: 1. Use the AutoGen framework to create user agents and assistant agents, and the agents collaborate to control SEM.

[0066] The user agent is used to identify user instructions and obtain the description of the region of interest from the user instructions. During the image capture process, the user agent calls SAM to segment the image, and then completes the positioning of the region of interest and triggers the image acquisition action. The user agent also uses the "Scanning Electron Microscope Instructions" and "Scanning Electron Microscope Hardware Protocol" as a knowledge base, and calls the knowledge base content during the dialogue with the user. Set prompt words for the user agent: As a SEM image capture technician, you need to determine whether the user's shooting instructions are correct. The correct instructions need to include target sample information, shooting parameters, and regions of interest; the region of interest is necessary information, and the others are auxiliary information; if the information provided does not meet the above conditions, you need to be prompted to supplement.

[0067] The assistant agent is defined as someone who understands the "Scanning Electron Microscope User Manual" and "Scanning Electron Microscope Hardware Protocol" documents, and writes the code logic for function calls based on the understanding of the above documents. Call the scanning electron microscope protocol interface to complete the automated control of the hardware device, and control the device to perform corresponding operations, such as moving the sample stage, autofocusing, etc. Set prompt words for the assistant agent: first think about the meaning of the user's instructions, disassemble the instructions into execution actions according to the SEM API interface, execute the function call of the interface after confirmation, and return the result to the user agent.

[0068] 2. The user enters natural language (text / voice) to describe the SEM shooting requirements through the agent mode dialogue window in the SEM control software. The description should include the target sample information, shooting parameters and area of ​​interest. For example, the user enters: "I want to shoot the solder ball sample in the sample stage." The user agent will disassemble the "sample type" and "shooting instruction" information, and decompose the user's needs into: the sample type is solder ball, and the shooting instruction is to confirm to start shooting.

[0069] 3. The assistant agent calls the "Scanning Electron Microscope Instructions" and "Scanning Electron Microscope Hardware Protocol" knowledge bases and executes the plan through natural language output, for example: (1) Determine the connection status of the scanning electron microscope.

[0070] (2) Move the sample stage to the center position and take an image.

[0071] (3). Call the image segmentation model SAM to segment the image and classify the segmentation results.

[0072] (4) If there is a “Solder Ball Sample” in the classification result, execute the shooting process.

[0073] (5). Call the automatic signal search interface.

[0074] (6). Call the automatic brightness and contrast interface.

[0075] (7). Call the auto focus interface.

[0076] (8). Call the FOV calibration interface.

[0077] (9). Move the sample stage to the sample position.

[0078] (10) Take an image.

[0079] According to the above plan, the assistant agent calls the code generation capability of the large language model to complete the function call code writing of the scanning electron microscope hardware protocol and executes the code to complete the hardware control of the SEM.

[0080] 4. The user agent displays the acquired image in the dialog window. After the user views and confirms the acquired image, he can give specific instructions for the next step of image analysis. For example, the user inputs: Please analyze the number of "solder balls" in this image. The user agent calls SAM to segment the image. After receiving the image segmentation results, the user agent counts the results and gives feedback to the user in the dialog window, for example: "The image contains 13512 solder balls."

[0081] Thus, through the above steps, the user operates the SEM in a dialogue form through the GUI to automatically find the area of ​​interest in the sample, automatically set the shooting parameters, automatically take images and complete data analysis. Compared with the traditional SEM operation mode, the embodiment of the present application has the following significant advantages and effects: directly converting natural language instructions into equipment control actions reduces the time of manually setting parameters and improves the efficiency of SEM operation; reduces the technical threshold of SEM operation, and users do not need to deeply learn complex equipment operation processes, and can control SEM equipment through natural language, so that more non-professionals can use SEM; natural language interaction provides a more intuitive and friendly user experience, and users can focus more on the experiment itself rather than the operation details, which enhances the user experience; the integrated SAM module can quickly perform image analysis, provide accurate data analysis results, accelerate the scientific research process, and improve data processing speed and accuracy. Through the elaboration of the above advantages and effects, the embodiment of the present application not only improves the control efficiency and user experience of the SEM equipment, but also brings substantial convenience and innovative power to scientific research.

[0082] Figure 8 FIG. 8 is a block diagram showing a hardware configuration of an apparatus 800 that can implement an embodiment of the present application. Figure 8 As shown, the apparatus 800 may include a processor 801 and a memory 802. The processor is configured to execute program instructions, and the memory is configured to store the program instructions. When the program instructions are loaded and executed by the processor, the apparatus executes the method for controlling an imaging device according to any one of the above embodiments. Figure 8 In the device 800, only the components related to this embodiment are shown. Therefore, it is obvious to those skilled in the art that the device 800 may also include Figure 8 The specific functions implemented by the memory 802 and the processor 801 of the device 800 provided in the embodiments of this specification can be explained in comparison with the aforementioned embodiments in this specification, and can achieve the technical effects of the aforementioned embodiments, so they will not be repeated here.

[0083] The apparatus 800 may correspond to a computing device having various processing functions. For example, the apparatus 800 may be implemented as various types of devices, such as a personal computer (PC), a server device, a mobile device, etc.

[0084] The processor 801 can control the operation of the device 800. For example, the processor 801 controls the operation of the device 800 by executing the program stored in the memory 802 on the device 800. The processor 801 can be implemented by a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), an artificial intelligence processor chip (IPU), etc. provided in the device 800. However, the present application is not limited to this. In this embodiment, the processor 801 can be implemented in any appropriate manner. For example, the processor 801 can take the form of a computer-readable medium, a logic gate, a switch, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a programmable logic controller, and an embedded microcontroller, etc., such as a microprocessor or a processor and a computer-readable program code (such as software or firmware) that can be executed by the (micro) processor.

[0085] The memory 802 may be used to store hardware of various data and instructions processed in the device 800. For example, the memory 802 may store processed data and data to be processed in the device 800. The memory 802 may store data that has been processed or to be processed by the processor 801, for example, natural language instructions input by a user, etc. In addition, the memory 802 may store applications, drivers, etc. to be driven by the device 800. For example: the memory 802 may store various programs related to the method of controlling an imaging device to be executed by the processor 801, etc. The memory 802 may be a DRAM, but the present application is not limited thereto. The memory 802 may include at least one of a volatile memory or a non-volatile memory. The non-volatile memory may include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a phase change RAM (PRAM), a magnetic RAM (MRAM), a resistive RAM (RRAM), a ferroelectric RAM (FRAM), etc. The volatile memory may include dynamic RAM (DRAM), static RAM (SRAM), synchronous DRAM (SDRAM), PRAM, MRAM, RRAM, ferroelectric RAM (FeRAM), etc. In an embodiment, the memory 802 may include at least one of a hard disk drive (HDD), a solid state drive (SSD), a high-density flash memory (CF), a secure digital (SD) card, a micro secure digital (Micro-SD) card, a mini secure digital (Mini-SD) card, an extreme digital (xD) card, caches, or a memory stick.

[0086] The specific functions implemented by the memory 802 and the processor 801 of the device 800 provided in the implementation mode of this specification can be explained in comparison with the aforementioned implementation modes in this specification, and can achieve the technical effects of the aforementioned implementation modes, and will not be repeated here.

[0087] It should be noted that, for the purpose of simplicity, the present application describes some methods and embodiments thereof as a series of actions and combinations thereof, but those skilled in the art will appreciate that the scheme of the present application is not limited by the order of the actions described. Therefore, based on the disclosure or teaching of the present application, those skilled in the art will appreciate that some of the steps therein may be performed in other orders or simultaneously. Further, those skilled in the art will appreciate that the embodiments described in the present application may be regarded as optional embodiments, i.e., the actions or modules involved therein are not necessarily necessary for the implementation of one or some schemes of the present application. In addition, depending on the different schemes, the present application also has different emphases on the description of some embodiments. In view of this, those skilled in the art will appreciate that the parts that are not described in detail in a certain embodiment of the present application may also refer to the relevant descriptions of other embodiments.

Claims

1. A method for controlling an imaging device, characterized in that: include: Acquire an operation instruction for the imaging device, where the operation instruction is used to instruct the imaging device to perform a target operation; The operation instruction is converted into operation information for the imaging device by a first conversion module, so as to determine the target operation that the imaging device is instructed to perform; The second conversion module converts the operation information into a control program for the imaging device according to the operation document of the imaging device, so as to control the imaging device.

2. The method according to claim 1, characterized in that Obtaining an operation instruction for the imaging device, including: Obtain user behavior instructions; In response to extracting specific information associated with the imaging device from the behavior instruction, an operation instruction for the imaging device is acquired from the behavior instruction.

3. The method according to claim 2, characterized in that The behavior instructions include natural language instructions and gesture instructions.

4. The method according to claim 2, characterized in that The specific information includes at least one of the following: sample information, operation parameters and target objects.

5. The method according to claim 2, characterized in that Converting the operation instruction into operation information for the imaging device includes: The operation instruction is converted into operation information for the imaging device according to the specific information.

6. The method according to claim 1, characterized in that Converting the operation information into a control program for the imaging device includes: Converting the operation information into an operation process for the imaging device, wherein the operation process is used to plan the action of the imaging device; The operation flow is converted into a control program for the imaging device.

7. The method according to claim 6, characterized in that Further including: In response to the operation instruction including an image acquisition instruction for the target object, the operation information is converted into an operation flow including the following contents: A first sub-operation process is used to control the imaging device to acquire a first image; A second sub-operation process is used to locate a region of interest in the first image; as well as The third sub-operation process is used to control the imaging device to acquire a second image associated with the region of interest.

8. The method according to claim 7, characterized in that The first sub-operation process includes: Adjusting the intrinsic parameters of the imaging device to obtain adjusted parameters; adjusting the position of the target object in the field of view of the imaging device to obtain an adjusted position; and Based on the adjustment parameter and the adjustment position, the imaging device is controlled to acquire a first image.

9. The method according to claim 8, characterized in that Adjusting the position of the target object in the field of view of the imaging device includes: The position of the target object is adjusted and / or the field of view of the imaging device is adjusted.

10. The method according to claim 8, characterized in that Based on the adjustment parameter and the adjustment position, controlling the imaging device to acquire a first image includes: Based on the adjustment parameters and the adjustment position, controlling the imaging device to acquire a sample image; Until the quality of the sample image is qualified, the following steps are cyclically performed: updating the intrinsic parameters of the imaging device to obtain updated parameters; updating the position of the target object in the field of view of the imaging device to obtain an updated position; and controlling the imaging device to acquire a sample image based on the updated parameters and the updated position; The sample image is used as the first image.

11. The method according to any one of claims 8 to 10, characterized in that: The intrinsic parameters include at least one of the following: magnification, brightness, contrast, signal-to-noise ratio, acceleration voltage, electron beam current, and focusing parameters.

12. The method according to claim 7, characterized in that The second sub-operation process includes: Segmenting the target object from the first image to obtain a target area corresponding to the target object; The target region is determined as a region of interest.

13. The method according to claim 7, characterized in that The third sub-operation process includes: Taking the region of interest as the center of the field of view, the imaging device is controlled to acquire a second image.

14. The method according to claim 1, wherein: The second conversion module includes an artificial intelligence model; through the second conversion module, according to the operation document of the imaging device, the operation information is converted into a control program for the imaging device, including: A model knowledge base is established based on the operation document of the imaging device, and the model knowledge base is used to guide the artificial intelligence model to convert the operation information into a control program for the imaging device.

15. The method according to claim 14, characterized in that The operation document includes: Operations for the imaging device and application programming interfaces corresponding to the operations.

16. A device, characterized in that: include: a processor configured to execute program instructions; as well as A memory configured to store the program instructions, which, when loaded and executed by the processor, causes the apparatus to perform the method according to any one of claims 1 to 15.

Images