Fluorescence microscopy control system and device based on localized large language model agent

Abstract

The application discloses a fluorescence microscope imaging control system based on a localized large language model agent, comprising: a fluorescence microscope function library, including a hardware library for controlling physical components of a fluorescence microscope and a software library for analyzing and processing fluorescence microscope images; a tool card, which integrates fluorescence microscope operations into tasks at different levels according to complexity; an agent, which uses a localized large language model to analyze user instructions, calls tasks in the tool card, and controls the fluorescence microscope to perform corresponding microscope operations through the hardware library and / or the software library. The application also discloses a fluorescence microscope imaging device based on a localized large language model agent, comprising a fluorescence microscope and a fluorescence microscope imaging control system in communication with the fluorescence microscope. The application reduces the pressure on local large model code generation, greatly improves the quality of code generation, and ensures data security and controllability, and stable system operation.

Description

Technical Field

[0001] This invention relates to the field of automation control technology, and in particular to a fluorescence microscopy imaging control system and device based on a localized large language model intelligent agent. Background Technology

[0002] In recent years, large language models, relying on their powerful language understanding and generation capabilities, have been widely applied in scenarios such as knowledge question answering and information retrieval, code and assisted development, data processing and analysis, and business automation. By linking with external systems, they achieve a closed loop from understanding to execution. Especially in the field of automation control, large language models can translate natural language into machine-understandable code, and through communication with hardware, realize the control of experimental procedures and instruments, ultimately forming a fully automated unmanned scientific research platform.

[0003] Taking microscope control as an example, the large language model can understand the operator's natural language and automatically execute commands to complete long-term unattended experiments such as live cell tracking. Therefore, researchers no longer need to be physically present at the microscope and can perform more valuable work such as data analysis and experimental design. Furthermore, due to the high consistency of automated operation, there is no cognitive bias inherent in human operation, and each operation is executed strictly according to the set parameters. With the help of the large language model, the microscope is no longer just an experimental tool, but has evolved into an intelligent experimental assistant for the operator.

[0004] Chinese patent document CN117672222A discloses a microscope control method driven by a large language model. The operator issues commands via voice, utilizes prompting engineering to define the large language model ChatGPT role, and incorporates software and hardware libraries as context. The text-to-speech transcription serves as input, generating and executing code to control a bright-field microscope. Specific scenarios include control of the stage, objectives, illumination, exposure, focus, and image analysis. Chinese patent document CN119211704A discloses a method for controlling imaging equipment. For the parameter settings and hardware control of a scanning electron microscope, it employs a two-stage conversion: first, natural language is parsed into operational information; then, this information is used to generate execution code based on the device's API documentation.

[0005] The above-mentioned existing technologies have the following defects: (1) Directly using a large language model such as ChatGPT has the following problems, including API call fees, susceptibility to network latency and access restrictions, and the risk of data leakage. In addition, it is difficult to customize because it cannot be deeply fine-tuned for specific domains. (2) Directly calling a large model to generate code for a vertical domain can easily lead to problems such as parameter errors and missing steps. If the generated code has errors, it is necessary to feed back to the model for retry, which leads to a longer call chain and low efficiency. (3) Since the cloud model has not been customized, its output is strongly affected by input prompts. This is the "dependence" on prompts. Even in a relatively general and simple scenario, hundreds of words are often needed as prompts. This not only requires developers to invest a lot of effort in maintenance, but also consumes more GPU memory during the inference process due to the long prompts. Summary of the Invention

[0006] This invention provides a fluorescence microscopy imaging control system based on a localized large language model intelligent agent, which has the ability to handle tasks of varying complexity.

[0007] The technical solution of the present invention is as follows: A fluorescence microscopy imaging control system based on a localized large language model intelligent agent includes: The fluorescence microscope function library includes a hardware library for controlling the physical components of the fluorescence microscope and a software library for analyzing and processing fluorescence microscopic images. The tool card integrates fluorescence microscope operations into different levels of tasks based on complexity, including basic and intermediate tasks; The intelligent agent uses a localized large language model to parse user commands, calls the tasks in the tool card, and controls the fluorescence microscope to perform corresponding microscope operations through the hardware library and / or software library.

[0008] Preferably, the hardware library includes: A fluorescence channel control module is used to provide excitation wavelengths and emission bands that match the fluorescent dye; The lighting control module is used to set the intensity of the bright field light source and the excitation light source. The stage control module is used to control the movement of the stage in three dimensions; Objective lens control module, used to switch between objectives with different magnifications; The camera control module is used to control the camera's shooting and set shooting parameters; The serial communication module is used to establish communication with the fluorescence microscope and to send and receive control commands and status information. The instruction input module is used to obtain user input information.

[0009] Preferably, the software library includes: The cell nucleus detection module is used to automatically identify and segment cells and cell nuclei in microscopic images; The image quality assessment module is used to detect microscopic image quality problems and provide suggestions for shooting parameters. An autofocus module is used to determine the optimal focal plane during fluorescence imaging; The region of interest detection module is used to identify and locate the coordinates of regions containing cells.

[0010] Preferably, in the tool card: The basic task is to directly control the fluorescence microscope hardware, including setting the light source brightness, switching objectives, moving the stage, and switching fluorescence channels. The intermediate-level tasks are those that combine operations on hardware and software libraries, including autofocus, moving the slide to the center of the field of view, and capturing images.

[0011] The basic tasks include hardware operations such as setting the light source brightness, switching objectives, moving the stage, and switching fluorescence channels, involving the extraction and understanding of parameters such as objective magnification and fluorescence channel names from instructions. The intermediate tasks combine operations from the hardware and software libraries, with a fixed internal execution flow.

[0012] Preferably, the localized large language model is a DeepSeek model that is fine-tuned using a fluorescence microscope to manipulate relevant datasets.

[0013] DeepSeek models that utilize datasets related to fluorescence microscopy operations for fine-tuning include: Construct a training dataset that includes the conversion of low-complexity instructions to code and the mapping of high-complexity instructions to low-complexity instructions; The DeepSeek model is fine-tuned by using a low-rank adaptation technique, freezing the original weights of the pre-trained model, and introducing a low-rank matrix in the key layer to update the weights.

[0014] The training dataset used for fine-tuning the localized large language model consists of two parts: the first part is the conversion of low-complexity instructions to code, including matching the corresponding tools for the instructions and extracting the parameters needed to call the tools from the instructions; the second part is parsing high-complexity instructions and mapping them into low-complexity instructions through semantic understanding.

[0015] The aforementioned high-complexity instructions refer to instructions that contain biological terminology.

[0016] Based on the aforementioned training dataset, the DeepSeek model was fine-tuned using LoRA (Low-Rank Adaptation). The training process employed mixed-precision computation to improve computational efficiency and applied a dynamic learning rate adjustment strategy, updating parameters via gradient descent within a preset training epoch. This fine-tuning enabled the large language model to learn specialized vocabulary, tool definitions, and task logic, generating a target model adapted to specific tool usage scenarios.

[0017] Preferably, the fluorescence microscopy imaging control system based on a localized large language model intelligent agent further includes a localized speech recognition model for converting user voice commands into text commands.

[0018] Preferably, the localized speech recognition model is a Whisper model that is fine-tuned by manipulating relevant speech datasets using a fluorescence microscope.

[0019] Preferably, the intelligent agent uses a localized large language model to parse user commands, invokes tasks in the tool card, and performs corresponding microscope operations through the hardware library and / or software library, including: (i) At the discourse level, the complex pointers in the user instructions are parsed and converted into a set of operations corresponding to each operation object, resulting in result one; (ii) At the operational level, the operations corresponding to each operation object in Result 1 are broken down into a series of minimum executable steps to obtain Result 2; Traverse the smallest executable steps in result 2, and execute (iii) and (iv) for each smallest executable step; (iii) At the domain level, identify the high-complexity instructions in result 2, and use the pre-set domain knowledge to convert the high-complexity instructions into low-complexity instructions to obtain result 3; (iv) At the tool level, select the corresponding tool according to Result 3 and the tool card supported by the system, and extract the parameters required to call the corresponding tool from Result 3 to control the fluorescence microscope to perform the corresponding microscope operation.

[0020] This invention also provides a fluorescence microscopy imaging device based on a localized large language model intelligent agent, comprising: Fluorescence microscope; The fluorescence microscopy imaging control system communicates with the fluorescence microscope and controls the fluorescence microscope to perform corresponding microscope operations according to user instructions.

[0021] The fluorescence microscope includes a fluorescence microscope body and an electric platform, an electric microscope lens, an information acquisition camera, and an electric filter wheel connected to the fluorescence microscope body.

[0022] This invention employs a decoupling design between function libraries (software libraries, hardware libraries) and tool cards. Tool cards encapsulate task logic and call one or more function libraries. This design offers the following advantages: function libraries can be reused by different tool cards, while tool cards are isolated from specific implementations, facilitating function combination and expansion, and reducing maintenance costs. Tool cards can encapsulate complex tasks, internally calling multiple function libraries; users only need to call one tool card to achieve complex operations, reducing the complexity of user command parsing. By layering tools according to complexity, different user needs are met. Especially when handling complex requests, the large language model does not need to plan how to call multiple different basic tools; it only needs to identify the user's intent and call the corresponding basic / intermediate tool, simplifying the decision-making required by the large language model and improving the accuracy and reliability of execution.

[0023] This invention employs a two-round instruction parsing process. First, complex instructions are converted into equivalent simple instructions. Then, the simple instructions are used to generate the final code call. This design reduces parsing complexity. Since complex instructions often contain technical terms, directly mapping them to code is prone to errors. Converting them into simple instructions first simplifies the subsequent code generation logic. It also enhances scalability. When adding new tools or functions, only the mapping relationship between complex and simple instructions needs to be expanded; the underlying code parsing module does not need to be modified, allowing the system to quickly adapt to new requirements. Finally, it introduces professional knowledge support. By leveraging a knowledge base in a vertical domain, the system can understand professional expressions, improving its ability to comprehend professional semantics.

[0024] This invention constructs a dataset to fine-tune the large language model used, enabling the model to accurately complete instruction parsing and parameter mapping. While ensuring controllable computational resources, its general understanding and reasoning capabilities are limited due to its relatively small scale. Through fine-tuning, the model can better understand specialized expressions such as "apoptotic sample" and "multi-channel," reducing manual intervention and multiple rounds of interaction, and achieving the generation of correct tool call results in a single pass.

[0025] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) Users do not need to understand the details of microscope operation or memorize instructions and parameters; (2) It reduces the pressure of generating code for large local models and can greatly improve the quality of code generation; (3) Data security is controllable, the system operates stably, and it meets compliance and customization requirements. Attached Figure Description

[0026] Figure 1 This is a structural block diagram of the fluorescence microscopy imaging control system based on a localized large language model intelligent agent according to the present invention. Detailed Implementation

[0027] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the embodiments described below are intended to facilitate the understanding of the present invention and do not limit it in any way.

[0028] This invention provides a method, system, and device for controlling fluorescence microscopy using an agent-driven architecture built around a localized large language model (DeepSeek). Unlike existing language control methods, this invention utilizes toolcards and function libraries to handle tasks of varying complexity. Fine-tuning the model improves code output accuracy, reduces the need for multi-turn interactions and manual maintenance, resulting in higher overall development efficiency while maintaining the inherent security and stability of the local model. This invention is applicable to fluorescence microscopy applications such as apoptosis and necrosis detection, and as an auxiliary research tool for drug screening.

[0029] like Figure 1 As shown, the fluorescence microscopy imaging control system based on a localized large language model intelligent agent of the present invention includes: The fluorescence microscope function library includes a hardware library for controlling the physical components of the fluorescence microscope and a software library for analyzing and processing fluorescence microscopic images. Tool cards integrate fluorescence microscope operations into different levels of tasks based on complexity; The intelligent agent uses a localized large language model to parse user commands, calls the tasks in the tool card, and performs corresponding microscope operations through the hardware library and / or software library.

[0030] The construction method of the system of the present invention specifically includes the following steps: Step S1: Construct the hardware library in the fluorescence microscope function library, including the fluorescence channel control module, illumination control module, stage control module, objective lens control module, camera control module, serial communication module, and voice input module; Step S2: Construct the software library in the fluorescence microscope function library, including the cell nucleus detection module, image quality evaluation module, autofocus module, and region of interest detection module.

[0031] Step S3: Construct tool cards 3, integrating microscope operations into different levels of tool cards based on complexity to reduce repetitive and low-value operations. Tool cards include: basic tasks and intermediate tasks.

[0032] Basic tasks include setting the light source brightness, switching objectives, moving the stage, and switching fluorescence channels. Intermediate tasks include autofocusing, moving the slide to the center of the field of view, and taking images.

[0033] Step S4: Construct a dataset for fine-tuning the local large language model (DeepSeek); fine-tune the DeepSeek open-source model based on LoRA.

[0034] Step S5: Construct a dataset for fine-tuning the local large speech model (Whisper); fine-tune the Whisper open-source model based on LoRA.

[0035] Step S6: Build an agent, initialize the localized large language model and speech recognition model, and process user commands to form prompt information as input.

[0036] The above steps are used to construct a fluorescence microscope control agent based on a localized large language model.

[0037] Furthermore, the fluorescence microscope hardware library includes a fluorescence channel control module, an illumination control module, a stage control module, an objective lens control module, a camera control module, a serial communication module, and a voice input module.

[0038] Specifically, the fluorescence channel control module provides the corresponding excitation wavelength and emission band for the fluorescent dyes in the sample. Fluorescent dyes include FITC (fluorescein isothiocyanate), TRITC (tetramethylrhodamine isothiocyanate), and DAPI (4',6-diamidinyl-2-phenylindole). The excitation wavelength of FITC is 460-495 nm, and the emission wavelength is 510-550 nm; the excitation wavelength of TRITC is 528-553 nm, and the emission wavelength is 578-633 nm; the excitation wavelength of DAPI is 360-390 nm, and the emission wavelength is 435-485 nm.

[0039] The lighting control module is used to set the intensity of the bright field light source and the excitation light source.

[0040] The stage control module enables the stage to move in the x, y, and z axes.

[0041] The objective lens control module is used to switch between objectives with different magnifications.

[0042] The camera control module is used to control the camera's shooting and set the camera's exposure time and gain.

[0043] The serial communication module allows the computer to establish communication with the microscope via a serial port. It sends control commands according to the instruction set defined by the microscope manufacturer and receives and parses the status information returned by the microscope.

[0044] The language input module is used to obtain text information from the operator.

[0045] Furthermore, the fluorescence microscope software library includes modules for cell nucleus detection, image quality assessment, autofocus, and region of interest detection.

[0046] Specifically, the cell nucleus detection module uses open-source deep learning models (such as Cellpose) to automatically identify and segment cells and cell nuclei in microscopic images, thereby completing cell localization, quantity statistics, and related morphological analysis.

[0047] The image quality assessment module detects image quality issues: overexposure, underexposure, and low signal-to-noise ratio (SNR). Overexposure detection determines if the image contains an excessively high proportion of extremely bright pixels. The system defines pixels with a brightness greater than or equal to 250 as overbright; if these pixels comprise more than 1% of the total image pixels, it is considered overexposure. Underexposure detection determines if the overall brightness of the image is too low. The system calculates the 90th percentile pixel value as the brightness level of the "brighter parts" of the image. If this value is below 100, the entire image is considered too dark. Overexposure and underexposure are detected by statistically analyzing the ratio of bright to dark pixels. Traditional image processing algorithms are used for image preprocessing, Otsu's binarization, morphological post-processing to segment cells and background, and SNR calculation. The formula for calculating SNR is: ; in, The average gray value of the signal region. The average gray value of the background area. The standard deviation of the background area is given. Shooting parameter suggestions are provided based on image quality analysis. For overexposure and underexposure issues, the exposure time and gain are reduced or increased respectively; for low signal-to-noise ratio issues, such as an SNR of less than 2 in the original image, the laser intensity is appropriately increased.

[0048] During fluorescence imaging, the autofocus module scans and acquires multiple candidate focal planes, calculates the standard deviation of image pixel intensity to evaluate image information content, and selects the focal plane with the largest standard deviation as the optimal focal plane. The formula for calculating the standard deviation is: ; Where N represents the total number of pixels in the image. Let be the grayscale value of the i-th pixel. This represents the average grayscale value of all pixels in the entire image.

[0049] The Interest Detection (OPI) module acquires images of the slide at equal intervals. Each acquired image is then processed by a cell nucleus detection module to extract and filter the coordinates of regions containing cells, which are then used as target points. By precisely locating cells, this can be used for automated navigation, rapid retrieval of regions of interest, and more.

[0050] Furthermore, the tool cards include: basic tasks, which directly manipulate microscope hardware, such as switching objectives and fluorescence channels, involving the extraction and understanding of parameters such as objective magnification and fluorescence channel names from instructions; intermediate tasks, which combine operations from hardware and software libraries, such as image capture, involving switching objectives, adjusting illumination, moving the stage, autofocusing, and capturing images, involving the extraction and understanding of parameters such as slide index, number of images, and imaging channels from instructions, while also supporting image quality assessment, adjusting exposure time and gain in cases of overexposure or underexposure, increasing laser intensity or reselecting the field of view in cases of low signal-to-noise ratio, until a specified number of high-quality images are completed; another example is locating the center of the field of view, involving region of interest detection and stage movement. The main internal processes of this level of task are already fixed, such as the fixed steps in the imaging task of turning off the background light, turning on the excitation light, and turning off the excitation light promptly after imaging to avoid photoquenching and photobleaching, these processes cannot be modified within the tool.

[0051] Further, in step S4, DeepSeek's medium-sized open-source instruction fine-tuning model with 7 billion parameters is used as the base model. This model is sufficient to handle complex inference, creation, and analysis tasks, and can be efficiently inferred and deployed on consumer-grade GPUs, making it very user-friendly for individual developers, research institutions, and small and medium-sized enterprises. Furthermore, this model is licensed under the Apache 2.0 license, meaning any individual or company can download, use, modify, and commercially deploy it without any copyright issues. In this invention, the model is downloaded and fine-tuned using data related to microscope operations.

[0052] The dataset used for fine-tuning contains approximately 1000 data pairs, mainly comprising two parts. The first part involves the conversion of low-complexity instructions into code, including matching instructions with appropriate tools, for example: User request: Switch to TRITC mode; Tool selection input: Given the tool descriptions as follows, {'Fluorescence_Imager_Tool':'Capture fluorescence images','Channel_Switcher_Tool':'Switch fluorescence channels'...}, select the tool to complete the following task: [User Request]; Tool selection output: Channel_Switch_Tool; This step actually provides the complete tool definition, as well as the parameters needed to call the tool from the instructions, for example: User request: Acquire 5 images from the DAPI channel of slide number 1; Input parameter extraction: Provide the tool name Fluorescence_Imager_Tool and the tool description {'Tool description': 'Tool for capturing fluorescence images', 'Input type': {'Number of images': 'Integer-Number of images', 'Task': 'String-Channel('FITC', 'TRITC', 'DAPI')'}}; Parameter extraction output: image_numbers=5, task='DAPI'; The second part involves parsing high-complexity instructions (instructions containing biological terminology, such as "cell nucleus"). This means that high-level natural language instructions with biological semantic representations input by the user are mapped into low-complexity instructions through semantic understanding, making them tasks that the tool can recognize and execute. For example: User request: Take an image of a cell nucleus on slide one; Input: It is known that imaging cell nuclei requires the DAPI channel; ...Rewrite the user's requirements [user request] into operation steps; Output: Take a DAPI channel image of slide one.

[0053] After obtaining a pre-trained large language model, the DeepSeek model is fine-tuned using LoRA (Low-Rank Adaptation) based on this training set. This technique freezes the original weights of the pre-trained model and only introduces low-rank matrices to update the weights in key layers, which can significantly reduce resource consumption and is an efficient and convenient fine-tuning method. The training process employs mixed-precision computation to improve computational efficiency and applies a dynamic learning rate adjustment strategy, updating parameters through gradient descent within a preset training epoch. Through fine-tuning, the large language model learns specialized vocabulary, tool definitions, and task logic, generating a target model adapted to specific tool-calling scenarios. An operator inputs the tools and parameters required to complete the task to construct a test set. On this test set, the prediction function call of the method of this invention is consistent with the reference call provided by the operator.

[0054] Further, in step S5, the Whisper open-source model uses OpenAI's Whisper-Large-V3 as the base model, and performs microscopy-related fine-tuning on this model. A pre-trained large speech model is obtained, and a low-rank adaptation module (LoRA) is configured for fine-tuning. The dataset uses the open-source speech synthesis model Cosy-Voice to synthesize a training set containing the target instructions; the input is synthesized speech, and the output is the corresponding operation text. On a self-built test set, the method of this invention reduces the baseline speech word error rate from 10.18% to 4.63%.

[0055] Further, in step S6, construct the agent, which involves calling the Deepseek model to complete the corresponding task according to the following steps: (1) At the discourse level, the target slide set is parsed, and expressions containing plural indices such as "these four slides", "each slide", and "the first and third slides" are converted into a set of operations corresponding to each slide index. For example, "take one image of each of these four slides" is rewritten as "take one image of the first slide; take one image of the second slide; take one image of the third slide; take one image of the fourth slide". The input provided to the Deepseek model in this step includes: (a) Task: Analyze the user request. If it includes multiple slide operations, break it down into specific steps for each slide; (b) Complete set, slide indexes such as [first slide, second slide, third slide, fourth slide]; (c) Rules: Check if words such as "all", "all", and "each" exist. If they exist, rewrite the request to apply actions and parameters separately for each slide and return a semicolon-separated string; (d) [User request], this step ends and the result one is returned. (2) At the operational level, each instruction in Result 1 is further decomposed, i.e., whether the user instruction contains multiple atomic instructions and they are split. By providing the Deepseek model with the tools currently supported by the Deepseek model and some commonly used conjunctions and discourse markers, the Deepseek model is helped to decompose the instructions. The inputs provided to the Deepseek model in this step include: (a) Task: split the user instruction into a series of minimum executable steps; (b) Complete set: slide index, objective magnification, fluorescence channel; (c) Rules: standard actions: switch objective, switch fluorescence channel, move stage, autofocus, take picture, and return a string; (d) [Result 1], this step ends and Result 2 is returned; (3) At the domain level, if the system detects that Result 2 contains a biological concept, such as "cell nucleus", then the instruction is considered a complex instruction. The system will load domain and methodological knowledge for various experiments to guide the Deepseek model to use this knowledge to transform the instruction into an experimental operation, i.e., a low-complexity instruction. Otherwise, Result 2 will be directly stored as Result 3. The input provided to the Deepseek model in this step includes: (a) Task: Interpret the user task in natural language; (b) Domain knowledge: For example, imaging the cell nucleus requires a DAPI channel; (c) [Result 2], this step ends, and Result 3 is returned; (4) At the tool level, the system selects a tool based on the currently supported tool cards and the corresponding tool descriptions. The inputs provided to the Deepseek model in this step include: (a) Task: Select a tool to complete the task; (b) System-supported tool descriptions, such as Fluorescence_Imager_Tool: capture fluorescence images, Auto_Focuser_Tool: autofocus; (c) [Result 3], the step ends and result 4a is returned; After determining the tool, the system constructs a complete tool call based on result 4a and the corresponding parameter description. The inputs provided to the Deepseek model in this step include: (a) Task: Extract parameters based on the tool name and description; (b) Rules: The return format is parameter=value, separated by commas; (c) Description of the corresponding tool, including tool function and parameters, for example, Objective_Changer_Tool: {Tool description: Change objective, Input type: {Objective: string – ('4x', '10x', '20x', '40x')}}.

[0056] The method of the present invention specifically includes the following steps: starting the microscope, initializing the system, and collecting user commands via voice or text; the system receives user commands, analyzes the requirements, selects tools, extracts and generates parameters to enable tool invocation.

[0057] The specific implementation of the system equipment of the present invention includes the following: a microscope system, comprising a microscope body, an electric object-carrying platform, an electric objective lens, an information acquisition camera, and an electric filter wheel.

[0058] The method and system of this invention can implement the invocation of individual tools or combinations of tools, and can support any task that can be parsed as a tool card. For example, quickly take a preview image of the second slide using a 4x scope; take a picture of this area using a 20x scope; take 10 pictures of the first slide using a 40x scope; open the TRITC channel and switch to the 40x objective lens; move to the first slide to find the field of view and then switch to 20x; only switch to the DAPI channel, no need to take a picture; place the third slide in the center of the field of view; if the previous picture wasn't good, refocus and take another picture; return to the 4x scope, and then move the fourth slide over to prepare; view the second slide using a 20x scope, but don't take a picture yet. The modules and their invocation order required to complete a specific task are provided by Deepseek when parsing the user request.

[0059] Example 1 This embodiment uses a user's voice request to "take a picture of the cell nucleus on slide 2" as an example. The workflow of the fluorescence microscopy imaging control intelligent agent system of the present invention is as follows: (1) Discourse level (understanding instructions) The user's voice request, "Take a picture of the cell nucleus on slide 2," was recognized as text by Whisper. DeepSeek determined that this command was singular and applied to slide 2. If the user said "all slides," it would automatically break it down into individual commands for each slide.

[0060] (2) Operational level (disassembly steps) DeepSeek breaks this high-level instruction down into a series of atomic instructions (minimum execution steps): Step 1: Move slide 2 to the center of the field of view by moving the stage; Step 2: Take an image of the cell nucleus.

[0061] (3) Domain level (professional knowledge transfer) No biological concept was detected in step 1, so this step is skipped.

[0062] (4) Tool level (calling tools) For the task of "moving slide 2 to the center of the field of view by moving the stage", select a tool. The agent selects "Centering_Locator_Tool" from the tool card and extracts the parameter slide_index=2 (the second slide) from the command.

[0063] (5) Function execution (task execution) The system executes the Centering_Locator_Tool. The Centering_Locator_Tool controls the stage to move the second slide to the center of the field of view.

[0064] (6) Domain level (professional knowledge transfer) DeepSeek detects the presence of the biological concept "cell nucleus" in step 2. It then uses its internal knowledge base, "Imaging the cell nucleus requires the use of the DAPI channel," to semantically map the "capture an image of the cell nucleus" step 2 into a lower-complexity, more specific operation: "capture an image of the DAPI channel."

[0065] (7) Tool level (calling tools) For the task "Take an image of the DAPI channel", select a tool. The agent selects "Fluorescence_Imager_Tool" and extracts the parameters image_numbers=1, task='DAPI'.

[0066] (8) Function execution (task execution) The system executes the tool Fluorescence_Imager_Tool. Fluorescence_Imager_Tool automatically completes the following steps: setting the correct lighting, selecting the appropriate objective lens, autofocusing, taking the picture, evaluating the quality of the captured images, and continuing until a specified number of high-quality images are captured. Finally, the cell nucleus detection module automatically counts the number of cells in the field of view to be 34.

[0067] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A fluorescence microscopy imaging control system based on a localized large language model intelligent agent, characterized in that, include: The fluorescence microscope function library includes a hardware library for controlling the physical components of the fluorescence microscope and a software library for analyzing and processing fluorescence microscopic images. The tool card integrates fluorescence microscope operations into different levels of tasks based on complexity, including basic tasks and intermediate tasks. The basic tasks are those that directly control the fluorescence microscope hardware, including setting the light source brightness, switching objectives, moving the stage, and switching fluorescence channels. The intermediate tasks are those that combine operations from the hardware library and software library, including autofocus, moving the slide to the center of the field of view, and taking images. The intelligent agent parses user commands using a localized large language model, invokes tasks from the tool card, and controls the fluorescence microscope to perform corresponding microscope operations through the hardware and / or software libraries, including: (i) At the discourse level, the complex pointers in the user instructions are parsed and converted into a set of operations corresponding to each operation object, resulting in result one; (ii) At the operational level, the operations corresponding to each operation object in Result 1 are broken down into a series of minimum executable steps to obtain Result 2; Traverse the smallest executable steps in result 2, and execute (iii) and (iv) for each smallest executable step; (iii) At the domain level, identify the high-complexity instructions in result 2, and use the pre-set domain knowledge to convert the high-complexity instructions into low-complexity instructions to obtain result 3; (iv) At the tool level, select the corresponding tool according to Result 3 and the tool card supported by the system, and extract the parameters required to call the corresponding tool from Result 3 to control the fluorescence microscope to perform the corresponding microscope operation.

2. The fluorescence microscopy imaging control system based on a localized large language model intelligent agent according to claim 1, characterized in that, The hardware library includes: A fluorescence channel control module is used to provide excitation wavelengths and emission bands that match the fluorescent dye; The lighting control module is used to set the intensity of the bright field light source and the excitation light source. The stage control module is used to control the movement of the stage in three dimensions; Objective lens control module, used to switch between objectives with different magnifications; The camera control module is used to control the camera's shooting and set shooting parameters; The serial communication module is used to establish communication with the fluorescence microscope and to send and receive control commands and status information. The instruction input module is used to obtain user input information.

3. The fluorescence microscopy imaging control system based on a localized large language model intelligent agent according to claim 1, characterized in that, The software library includes: The cell nucleus detection module is used to automatically identify and segment cells and cell nuclei in microscopic images; The image quality assessment module is used to detect microscopic image quality problems and provide suggestions for shooting parameters. An autofocus module is used to determine the optimal focal plane during fluorescence imaging; The region of interest detection module is used to identify and locate the coordinates of regions containing cells.

4. The fluorescence microscopy imaging control system based on a localized large language model intelligent agent according to claim 1, characterized in that, The localized large language model is a DeepSeek model that is fine-tuned using relevant datasets manipulated by fluorescence microscopy.

5. The fluorescence microscopy imaging control system based on a localized large language model intelligent agent according to claim 4, characterized in that, DeepSeek models that utilize datasets related to fluorescence microscopy operations for fine-tuning include: Construct a training dataset that includes the conversion of low-complexity instructions to code and the mapping of high-complexity instructions to low-complexity instructions; The DeepSeek model is fine-tuned by using a low-rank adaptation technique, freezing the original weights of the pre-trained model, and introducing a low-rank matrix in the key layer to update the weights.

6. The fluorescence microscopy imaging control system based on a localized large language model intelligent agent according to claim 1, characterized in that, It also includes a localized speech recognition model for converting user voice commands into text commands.

7. A fluorescence microscopy imaging device based on a localized large language model intelligent agent, characterized in that, include: Fluorescence microscope; The fluorescence microscopy imaging control system according to any one of claims 1-6 communicates with the fluorescence microscope and controls the fluorescence microscope to perform corresponding microscope operations according to user instructions.

8. The fluorescence microscopy imaging device based on a localized large language model intelligent agent according to claim 7, characterized in that, The fluorescence microscope includes a fluorescence microscope body and an electric platform, an electric microscope lens, an information acquisition camera, and an electric filter wheel connected to the fluorescence microscope body.

Images