Plant phenotype analysis intelligent agent system based on large language model

Abstract

The application discloses a plant phenotype analysis intelligent agent system based on a large language model. The system adopts a star topological structure, takes a planning intelligent agent as a hub, and is provided with special intelligent agents for vision, model fine tuning, code generation, GWAS analysis and paper writing, so that an overall process from multi-scale plant image processing, zero sample plant image segmentation, plant three-dimensional reconstruction and point cloud segmentation, neural network model optimization, visual chart drawing, genotype-phenotype analysis to scientific research paper writing is realized, and a one-stop intelligent solution is provided for accelerating high-throughput phenomics and genomics correlation research.

Description

Technical Field

[0001] This invention belongs to the fields of crop phenotyping, agricultural automation, and artificial intelligence, and specifically relates to an intelligent agent system for plant phenotyping based on a large language model. Background Technology

[0002] With the increasing severity of global climate change and food security issues, how to efficiently acquire and analyze crop phenotypic information has become a key scientific problem in modern breeding and precision agriculture. Crop phenotype is not only a direct reflection of the interaction between genotype and environment, but also an important foundation for revealing crop growth mechanisms and assessing stress resistance and yield potential. In recent years, with the rapid development of multi-source sensing technology, various sensing devices, from mobile phones and gantry cranes to drones, can collect multimodal data at different spatial scales, including crop organs, individual plants, and even population levels. This data provides new opportunities for refined phenotypic analysis, but also brings challenges such as significantly increased data heterogeneity and analytical complexity.

[0003] On the one hand, data acquired by different sensors exhibit significant differences in spatial resolution, illumination conditions, and geometric features, making cross-device, cross-scale, and cross-modal phenotypic analysis tasks highly complex. Training dedicated deep learning models for specific crop types and imaging conditions has proven to be an effective way to obtain high-precision phenotypic analysis results. However, in practical applications, when the analysis object, imaging conditions, or research objectives change, the selection, combination, training, and adaptation of models often rely on researchers' human experience and manual configuration, lacking a unified decision-making and scheduling mechanism, making it difficult to efficiently respond to the frequently changing analytical needs in real agricultural scenarios. On the other hand, the design and optimization process of neural network models highly depends on researchers' professional knowledge, including network structure design, hyperparameter tuning, and training strategy selection. This human-centered approach to model building and adjustment is not only costly and time-consuming but also makes it difficult for phenotypic analysis models to be quickly transferred and reused across different crops, tasks, and data distributions, becoming a significant bottleneck restricting the large-scale application of intelligent agricultural systems.

[0004] More importantly, current phenotypic analysis workflows still suffer from systemic fragmentation. Researchers often need to complete multiple steps separately, such as literature review, data preprocessing, model training, two-dimensional or three-dimensional phenotypic extraction, and gene association analysis (GWAS). These steps lack a unified integration and scheduling framework, leading to low research efficiency and hindering the efficient reuse of knowledge and the expansion of functional modules. Therefore, constructing a comprehensive phenotypic analysis framework that can uniformly schedule multi-source data processing, model analysis, and optimization processes has become an urgent need in the field of intelligent crop phenotypic analysis. Summary of the Invention

[0005] (a) Technical problems to be solved The technical problems to be solved by this invention are as follows: First, existing plant phenotyping technologies are mostly limited to single data types and single analysis steps, lacking integrated system processing capabilities for multi-scale data such as microscopic cells, single plant organs, and field populations, as well as multi-modal data such as two-dimensional images, three-dimensional point clouds, and genotypes, and thus cannot cover the research needs of the entire phenotyping analysis chain. Second, the selection, optimization, and fine-tuning of neural network models used in plant phenotyping analysis are highly dependent on human experience, with difficulties in model transfer adaptation, insufficient zero-sample segmentation capabilities, and a lack of dedicated functional modules that can autonomously complete model optimization. The level of intelligence and analytical accuracy are difficult to meet the needs of diverse agricultural scenarios. Third, research steps such as phenotyping parameter extraction, GWAS correlation analysis, data processing and chart drawing, and scientific paper writing are fragmented, lacking a unified scheduling and collaborative architecture centered on a large language model, resulting in fragmented, long-cycle, and inefficient research processes. Fourth, existing phenotyping systems are mostly simple stacks of functional modules, lacking a unified underlying inference engine, standardized interaction interfaces, and autonomous task planning and scheduling mechanisms. The semantics of each module are misaligned, and the interaction is not standardized, making it impossible to achieve fully automated, one-stop intelligent analysis from data input to output.

[0006] (II) Technical Solution To address the aforementioned problems, this invention provides the following technical solution: a plant phenotypic analysis intelligent agent system based on a large language model is proposed, the specific solution of which is as follows.

[0007] A plant phenotypic analysis intelligent agent system based on a large language model includes a multi-agent collaborative system. The multi-agent collaborative system uses a large language model as the core inference engine and integrates six dedicated intelligent agent modules: a planning intelligent agent module, a visual intelligent agent module, a model fine-tuning intelligent agent module, a code intelligent agent module, a GWAS analysis intelligent agent module, and a paper writing intelligent agent module. Each of the six dedicated intelligent agent modules is equipped with a corresponding toolbox and neural network model to perform its respective phenotypic analysis-related tasks.

[0008] The core functions of each agent module are defined as follows: The planning agent module is used to parse and analyze user needs based on user input prompts and multimodal data, plan the overall task execution process based on the analysis results, break down the overall task into multiple sub-tasks, and assign each sub-task to the corresponding dedicated agent module for sequential execution; it is also used to summarize and integrate the execution results of the entire agent system after all the planned sub-tasks have been executed by their respective dedicated agent modules, and feed back the processed final result to the user; The visual intelligent agent module communicates with the planning intelligent agent module and, when scheduled and executed by the planning intelligent agent module, performs image classification, image segmentation, and target detection processing operations based on the two-dimensional plant image data input by the user; performs three-dimensional reconstruction processing operations based on the two-dimensional plant multi-view image data input by the user; performs point cloud segmentation processing operations based on the three-dimensional plant point cloud data input by the user; and extracts the corresponding plant phenotypic parameters based on the above processing results. The model fine-tuning agent module communicates with the planning agent module and is used to fine-tune and optimize the neural network model in the visual agent module based on the user-provided labeled structured plant dataset when scheduled and executed by the planning agent module. This involves adjusting the parameters or model architecture of the neural network model to make it more suitable for the user-provided plant dataset. The code agent module communicates with the planning agent module. When scheduled for execution by the planning agent module, it automatically writes and executes computer code according to task requirements to complete data processing and customized chart drawing operations. If an error occurs or the task is not completed during execution, the computer code is corrected according to the error information and re-executed until the task assigned by the planning agent module is completed. The GWAS analysis agent module communicates with the planning agent module. When scheduled by the planning agent module, it performs population structure analysis and kinship matrix calculation based on the genotype data provided by the user and the plant phenotypic parameters output by the visual agent module. Finally, it draws Manhattan diagrams and QQ diagrams to discover significantly associated SNP loci. The paper writing intelligent agent module communicates with the planning intelligent agent module. When scheduled and executed by the planning intelligent agent module, it automatically generates and writes scientific research papers based on the multimodal data provided by the user and the processing results output by the multi-agent collaborative system. After the first draft of the paper is completed, the paper content is autonomously reviewed and iteratively optimized.

[0009] Furthermore, the multi-agent collaborative system adopts a star topology architecture, with the planning agent module serving as the core scheduling node. The remaining specialized agent modules act as execution nodes connected to the core scheduling node, with the planning agent module handling task allocation, instruction forwarding, and result aggregation. Each agent module uses the Qwen2.5 large language model as a unified underlying inference engine, relying on this model to achieve intent understanding, logical reasoning, decision planning, and content generation. The multi-agent collaborative system also constructs a unified standardized application interface for interactive operations such as task instruction issuance, execution status feedback, intermediate data transmission, and processing result reporting. Each specialized agent module is independently configured with a toolbox and neural network model matching its function, supporting specialized processing capabilities and achieving specialized division of labor and collaborative operation within a unified scheduling framework.

[0010] Furthermore, the planning agent module, as the core scheduling node of the system, specifically executes subtask allocation and scheduling according to the following rules: When the user prompt contains image processing information such as image classification, image segmentation, and phenotypic parameter extraction, and the input data is a single plant image, the planning agent module schedules the visual agent module to execute the plant image processing task. When the user prompt contains 3D reconstruction information and the input data is a multi-view image of a plant, the planning agent module schedules the visual agent module to perform the 3D reconstruction task of the plant point cloud. When the user prompt contains point cloud segmentation information and the input data is a three-dimensional point cloud of a plant, the planning agent module schedules the visual agent module to perform the three-dimensional point cloud segmentation task of the plant. When the user prompt contains optimization model information and the input data is a labeled structured plant dataset, the planning agent module schedules the model fine-tuning agent module to execute the neural network model optimization task; When the user prompt includes information on drawing charts and cleaning data, and the input data is in Excel and CSV format, the planning agent module schedules the code agent module to automatically write computer code and execute it to complete the task. When the user prompt contains gene analysis information and the input data is genotype data, the planning agent module schedules the GWAS analysis agent module to perform the task of calculating the relationship between genotype and phenotype, and draws a Manhattan diagram and a QQ diagram to visualize the results. When the user prompt includes information about writing a paper, the planning agent module schedules the paper writing agent module to execute the paper writing task.

[0011] Furthermore, when the visual intelligent agent module performs plant image classification, it is specifically implemented through the following units: The feature extraction unit is used to extract the feature vector f of the plant image using the DINOV3 pre-trained feature extractor in the toolbox. p Meanwhile, the visual intelligent agent module pre-sets feature vectors for various plants. m represents the number of plant feature vectors contained in the vector; Similarity matching unit, used to match feature vector f p Cosine similarity is calculated for each of the m feature vectors fi, resulting in m cosine similarity values. If C i The maximum value C in max If the value is greater than the preset threshold τ, then the matching between the user-input plant image and the preset plant image is completed, thus completing the image classification task; if C i The maximum value C in maxIf the value is less than the preset threshold τ, the language description classification unit is triggered; The language description classification unit integrates a BLIP pre-trained neural network model. When the similarity matching unit cannot complete the plant image classification task, the BLIP model describes the plant image input by the user in human natural language. Then, the big language model inference core performs named entity recognition on the natural language description to obtain the plant name in the natural language description and complete the plant image classification.

[0012] Furthermore, the visual intelligent agent module integrates three neural network models to address plant image processing tasks at different scales: the SegFormer model for plant cell microscopic image segmentation, the YOLO model for organ-level segmentation and detection counting of single plant images, and the DeepLabV3 model for UAV field plant image segmentation.

[0013] Furthermore, the visual intelligent agent module also integrates SAM and CLIP pre-trained neural network models, used to perform zero-shot image segmentation when the three neural network models cannot complete the plant image segmentation task or the segmentation results cannot meet user requirements. This is specifically implemented through the following units: The image preprocessing unit is used to perform standardized preprocessing on the input two-dimensional plant image, uniformly adjusting the image resolution to 512×512. Then, the Z-score normalization method is used to normalize the pixel values ​​of the RGB three channels of the image to eliminate the interference caused by differences in light intensity and shooting angle, while preserving the key phenotypic features of the plant. The candidate mask generation unit is used to input the preprocessed plant image into the SAM pre-trained model. Through the model's built-in sliding window anchor point generation mechanism, it generates a full-image-level candidate mask set at different scales of the image. The number of candidate masks is controlled to be 150-200. Each mask is represented by a binary matrix. The mask resolution is consistent with the input image. At the same time, the non-maximum suppression algorithm is used to initially filter redundant masks with an overlap of ≥70%, and finally obtains a candidate mask set of 45-60. The feature matching unit is used to input the plant image classification name as a prompt word into the text encoder of the CLIP model to generate a plant category text feature vector; at the same time, it inputs each candidate mask set generated by SAM into the image encoder of the CLIP model to extract the image feature vector of the mask region; by calculating the cosine similarity between the text feature vector and the feature vector of each mask image, the matching score between each candidate mask and the target plant category is obtained. The matching score is normalized to the [0,1] interval. The higher the score, the higher the confidence that the area corresponding to the mask is the target plant. The mask fusion unit is used to employ the Top-K adaptive selection algorithm, where the K value is equal to the rounded-up result of the total number of candidate masks × 0.3, to select the K high-confidence mask regions with the highest matching scores; all high-confidence masks are fused to obtain the complete plant target segmentation mask, thus completing the zero-shot plant image segmentation task.

[0014] Furthermore, the visual intelligent agent module incorporates the VGGT model and the PointNet++ model; the VGGT model is used for 3D reconstruction of plant point clouds, directly calculating the 3D point cloud model of the plant based on the multi-view image of the plant input by the user; the PointNet++ model is used for plant point cloud segmentation, directly completing the point cloud segmentation of plant organs based on the 3D point cloud of the plant input by the user.

[0015] Furthermore, the model fine-tuning agent module optimizes the parameters of the neural network model in the visual agent module, specifically through the following units: The hyperparameter optimization unit is used to perform a global search on key hyperparameters such as learning rate, weight decay and batch size using the Bayesian hyperparameter optimization algorithm to determine the optimal training configuration. The full training unit is used to perform full parameter training on the SegFormer, YOLO, and DeepLabV3 neural network models with appropriate parameter amounts in the visual agent module, based on the structured plant image dataset provided by the user, so as to enhance the adaptability of the neural network model to the dataset. The lightweight fine-tuning unit is used to fine-tune some parameters of the neural network model with a large number of SAM, CLIP and BLIP parameters in the visual agent module, based on the structured plant image dataset provided by the user, using the LoRA algorithm, so as to enhance the adaptability of the neural network model to the dataset. The model fine-tuning agent module is also used to optimize the model architecture of the neural network model in the visual agent module. Specifically, it adopts a teacher-student collaborative training strategy and combines a neural architecture search algorithm to search and optimize the internal structure of the neural network model in order to improve the execution accuracy of the visual agent module in plant image processing tasks.

[0016] Furthermore, the code agent module automatically writes and executes computer code. Specifically, within a secure Python sandbox, the toolbox of the code agent module automatically generates code using libraries including Pandas and Matplotlib to complete data processing and customized chart drawing tasks. The toolbox has introspective error tracking capabilities, used to read error messages when code execution fails, correct logical errors in the code, and re-execute the computer code until all assigned tasks are successfully completed.

[0017] Furthermore, the paper writing intelligent agent module specifically implements paper writing through the following units: The literature retrieval and knowledge base construction unit has a deep internet search function, which is used to search for academic literature and review articles in relevant fields from Google Scholar based on user prompts. Then, it uses retrieval enhancement generation technology to transfer the searched academic literature and review articles into the vector knowledge base of the writing agent. The paper writing unit is used to automatically complete the writing of scientific papers based on the processing results output by the multi-agent collaborative system. Among them, the plant phenotypic parameters obtained by the visual agent module and the GWAS analysis results obtained by the GWAS analysis agent module are used as the core data for paper analysis, and the customized charts generated by the code agent module are used for the visualization of the analysis results. The paper review and optimization unit is used to independently review and iteratively optimize the content of the paper after the initial draft is completed, until the scientific paper is completed.

[0018] (III) Beneficial Effects Compared with the prior art, the present invention has at least the following positive technical effects.

[0019] Existing plant phenotyping techniques are mostly limited to single data types and single analysis steps, only achieving local image segmentation, phenotypic extraction, or simple data processing. They lack integrated analytical capabilities for multi-scale data such as microscopic cells, single plant organs, and field populations, as well as multi-modal data such as 2D images, 3D point clouds, and genotypes. Furthermore, they suffer from weak zero-sample adaptation and poor model generalization, making it difficult to cover the full-scenario, full-chain phenotypic and gene association analysis needs. To address this issue, this invention proposes a plant phenotyping intelligent agent system based on a large language model. Through multi-agent collaboration, it achieves full-process coverage from multi-scale plant image processing, zero-sample image segmentation, 3D reconstruction and point cloud segmentation, to neural network model optimization, GWAS genotype-phenotypic association analysis, and research paper writing. The system can adaptively adapt to different crops, imaging conditions, and research objectives, achieving zero-sample segmentation without requiring large amounts of labeled data. Model accuracy and scenario adaptability are significantly improved, truly realizing the fusion analysis of phenomics and genomics data, providing a complete and refined intelligent analysis solution for complex agricultural breeding scenarios.

[0020] Existing plant phenotyping research workflows heavily rely on manual operation and professional experience. Model selection, hyperparameter tuning, code writing and debugging, data statistics, graphing, GWAS analysis, and paper writing are fragmented and executed step-by-step. This is not only time-consuming and labor-intensive, but also suffers from high subjectivity, repetitive work, long research cycles, and extremely high barriers to entry, completely failing to meet the high-efficiency research needs of modern high-throughput breeding and precision agriculture. This invention constructs a fully autonomous and collaborative intelligent agent system to achieve fully automated, unmanned operation from multimodal data input to scientific paper output: the planning agent uniformly schedules tasks, the code agent automatically writes, corrects, and executes code, the model fine-tuning agent autonomously optimizes parameters and architecture, and the paper writing agent automatically retrieves literature, writes, and iterates manuscripts. This entire solution significantly reduces the technical barriers and labor costs of phenotyping research, compressing the traditional research process of weeks or even months into highly efficient completion, significantly improving research efficiency and reproducibility, and providing strong technical support for high-throughput, large-scale plant phenotyping and genome-wide association studies.

[0021] Existing plant image segmentation technologies often require manual annotation of large amounts of sample data for model training, targeting different crops, growth stages, and imaging scenarios. This is not only labor-intensive, time-consuming, and costly, but also results in poor model adaptability when faced with unlabeled new crops or scenarios, hindering rapid deployment in practical scientific research. This invention, however, achieves zero-shot image segmentation by integrating SAM and CLIP pre-trained models. It eliminates the need for extensive manual annotation, enabling rapid segmentation of plant images across different crops and scenarios through simple plant category classification. This significantly reduces the labor and time costs of sample annotation while greatly improving the scenario adaptability and generalization ability of the segmentation technology. Furthermore, the segmentation accuracy of this zero-shot process meets core research needs such as plant phenotypic extraction and organ identification. Combined with the system's fully automated capabilities, this further enhances the efficiency and convenience of plant phenotypic analysis, providing crucial technical support for rapid phenotypic analysis of diverse crops in high-throughput breeding.

[0022] Existing phenotypic analysis systems are mostly simple stacks of functional modules, lacking a unified inference engine, standardized communication interfaces, and intelligent scheduling mechanisms aligned with scientific research principles. These systems suffer from semantic misalignment, non-standardized interactions, and missing decision-making logic, enabling only passive execution and failing to autonomously plan processes based on actual research needs. This results in a severe disconnect from the real-world research logic of plant phenotyping. This invention innovatively adopts a star topology architecture, using a large language model as the unified inference core and a planning agent as the central hub to complete task parsing, decomposition, allocation, and result aggregation. Standardized interfaces are designed to decouple, extend, and standardize communication among the agents. The system scheduling strategy deeply aligns with the research processes of phenotypic and gene association analysis, resulting in more rational task allocation, more efficient collaboration, and more stable interactions. It completely solves the pain points of traditional systems—fragmentation, chaotic scheduling, and poor scalability—possessing strong biological interpretability and research adaptability. This provides a reusable, scalable, and iterative standardized collaborative framework for the field of intelligent plant phenotyping. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the architecture of the intelligent agent system for plant phenotyping based on a large language model according to the present invention.

[0024] Figure 2 This is a flowchart illustrating the process of image segmentation of rice images using the plant phenotypic analysis intelligent agent system based on a large language model according to the present invention.

[0025] Figure 3 This is a flowchart illustrating the process of extracting phenotypic parameters from rice images using the intelligent agent system for plant phenotypic analysis based on a large language model, as described in this invention.

[0026] Figure 4 This is a before-and-after comparison of point cloud segmentation of three-dimensional point cloud data of cabbage by the intelligent agent system for plant phenotypic analysis based on a large language model according to the present invention. Detailed Implementation

[0027] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The core architecture of the plant phenotypic analysis intelligent agent system based on a large language model disclosed in this invention is as follows: Figure 1 As shown, the system includes a multi-agent collaborative system with a large language model as the core inference engine. The multi-agent collaborative system adopts a star topology architecture and integrates six dedicated agent modules. Each module is equipped with a toolbox and neural network model with corresponding functions. Under the unified scheduling of the planning agent module, the system collaboratively completes the entire process of plant phenotypic analysis.

[0028] I. System Overall Architecture In this embodiment, the multi-agent collaborative system is built using a star topology architecture. The planning agent module serves as the core scheduling node of the system, while the visual agent module, model fine-tuning agent module, code agent module, GWAS analysis agent module, and paper writing agent module all serve as execution nodes. A bidirectional communication link is established with the core scheduling node, and the planning agent module uniformly completes task allocation, instruction forwarding, status monitoring, and result summarization.

[0029] In this embodiment, all agent modules uniformly adopt the Qwen2.5-72B large language model as the underlying inference engine, with 24GB of inference memory and a context window length of 8K tokens. Relying on the powerful capabilities of the large language model, user intent understanding, logical reasoning, decision planning and content generation are realized, ensuring semantic alignment, standardized interaction and efficient collaboration among the agent modules.

[0030] In this embodiment, the system has a unified and standardized RESTful application interface, which defines the task instruction specification in JSON format, the execution status code standard, the intermediate data transmission format and the processing result reporting protocol. This enables decoupling design, modular expansion and standardized communication between the planning agent module and each execution agent module, ensuring the orderliness of task scheduling and the consistency of the collaborative process.

[0031] In this embodiment, each dedicated intelligent agent module is independently configured with a dedicated toolbox and neural network model that matches its function, carrying specialized processing capabilities. Under a unified scheduling framework, it achieves specialized division of labor and collaborative operation. The specific configuration is as follows: Visual intelligent agent module: integrates DINOV3, BLIP, SegFormer, YOLO, DeepLabV3, SAM, CLIP, VGGT, PointNet++ models, as well as corresponding image preprocessing, feature extraction, and phenotypic parameter calculation toolboxes; Model fine-tuning agent module: integrates Bayesian hyperparameter optimization tools, LoRA fine-tuning tools, teacher-student collaborative training tools, neural architecture search tools, and model training, validation, and performance evaluation toolkits; Code Intelligence Module: Integrates data processing and visualization code libraries including Pandas and Matplotlib, as well as toolkits for code generation, debugging, execution, and error tracking; GWAS Analysis Agent Module: Integrates genome-wide association analysis tools including PLINK and GCTA, as well as a toolbox for population structure analysis, kinship calculation, and visualization plotting; Thesis Writing Intelligent Agent Module: Integrates Google Scholar search interface, search enhancement generation tools, vector knowledge base, and toolkits for thesis writing, grammar correction, logical review, and formatting optimization; The planning intelligent agent module integrates a toolbox for task parsing, process planning, node scheduling, and result aggregation, serving as the system's central hub to achieve full-process control.

[0032] II. Specific Implementation Methods of Each Intelligent Agent Module 1. Planning Agent Module In this embodiment, the planning agent module serves as the core scheduling hub of the system, and performs its functions according to the following process: First, it receives prompts and multimodal data input by the user, performs deep analysis and semantic understanding of the user's needs through the underlying large language model, and clarifies the user's core research objectives, input data types, and expected output results; Second, based on the results of the needs analysis, it plans the execution flow of the overall task, decomposes the overall task into multiple sequentially executable sub-tasks, and clarifies the execution objectives, input data, and output requirements of each sub-task; Third, according to the preset scheduling rules, it assigns each sub-task to the corresponding dedicated agent module, and simultaneously issues task instructions and input data, and monitors the execution status of each agent module in real time; Fourth, after all sub-tasks have been completed, it collects the processing results output by each dedicated agent module, summarizes and integrates the execution data, intermediate results, and final results of the entire process, forms a complete task execution report, and feeds it back to the user.

[0033] In this embodiment, the subtask allocation and scheduling rules of the planning agent module are as follows: When the user prompt contains image processing information such as image classification, image segmentation, and phenotypic parameter extraction, and the input data is a single plant image, the visual intelligent agent module is scheduled to execute the plant image processing task. When the user prompt contains 3D reconstruction information and the input data is a multi-view image of a plant, the visual intelligent agent module is scheduled to perform the 3D reconstruction task of the plant point cloud. When the user prompt contains point cloud segmentation information and the input data is a 3D point cloud of a plant, the visual intelligent agent module is scheduled to perform the 3D point cloud segmentation task of the plant. When the user prompt contains information on optimizing the model and the input data is a labeled structured plant dataset, the scheduling model fine-tuning agent module executes the neural network model optimization task. When the user prompts include information on drawing charts and cleaning data, and the input data is in Excel and CSV format, the scheduling code agent module automatically writes computer code and executes it to complete the task. When the user prompt contains gene analysis information and the input data is genotype data, the GWAS analysis agent module is scheduled to perform the task of calculating the relationship between genotype and phenotype, and to draw Manhattan diagrams and QQ diagrams to visualize the results; When the user prompts include information about writing a paper, the paper writing agent module is scheduled to perform the paper writing task.

[0034] In this embodiment, the system uses files to uniformly store the historical dialogue records and all intermediate processing results of each intelligent agent module. The file content is updated in real time during task execution to ensure the consistency of content and data traceability of multi-step task execution. The planning intelligent agent module finally completes the summary and integration of the entire process results based on the file.

[0035] 2. Visual Intelligent Agent Module In this embodiment, when the visual intelligent agent module is scheduled and executed by the planning intelligent agent module, it performs corresponding image processing operations according to the input data type and task requirements, and extracts plant phenotypic parameters. The specific implementation method is as follows: (1) Implementation of plant image classification task When the user's requirement is to classify plant images, the visual agent module completes the classification task through a three-level unit: The feature extraction unit is used to extract the feature vector f of the plant image using the DINOV3 pre-trained feature extractor in the toolbox. p Meanwhile, the visual intelligent agent module pre-sets feature vectors for various plants. m represents the number of plant feature vectors contained in the vector; Similarity matching unit, used to match feature vector f p Cosine similarity is calculated for each of the m feature vectors fi, resulting in m cosine similarity values. If C i The maximum value C in max If the value is greater than the preset threshold τ, then the matching between the user-input plant image and the preset plant image is completed, thus completing the image classification task; if C i The maximum value C in max If the value is less than the preset threshold τ, the language description classification unit is triggered; The language description classification unit integrates a BLIP pre-trained neural network model. When the similarity matching unit cannot complete the plant image classification task, the BLIP model describes the plant image input by the user in human natural language. Then, the big language model inference core performs named entity recognition on the natural language description to obtain the plant name in the natural language description and complete the plant image classification.

[0036] (2) Implementation of plant image segmentation task In this embodiment, the visual intelligent agent module is configured with dedicated segmentation models for plant images at different scales, as detailed below: For the task of segmenting plant cell microscopic images, the SegFormer model is called to perform segmentation processing. For organ-level segmentation and detection counting tasks of single plant images, the YOLO model is invoked to perform segmentation and detection processing; Figure 2 The presentation demonstrates the workflow of organ-level segmentation of a single rice plant image. The dialog box on the right illustrates the interaction between the user and the multi-agent collaborative system: the user issues a task instruction to the system, with the core requirement being image segmentation and the input data being a single rice plant image; after receiving the instruction, the planning agent module first parses and breaks down the user's task, and then schedules the visual agent module to execute the specific segmentation task. This module uses a YOLO neural network model to complete the organ-level segmentation of the rice image and visualizes the segmentation results in the central area of ​​the interface; finally, the planning agent module summarizes the entire task execution process and results and provides feedback to the user with the final conclusion.

[0037] For the task of group segmentation of field plant images from UAVs, the DeepLabV3 model is called to perform segmentation processing.

[0038] When the above three specialized models fail to complete the plant image segmentation task, or the segmentation results do not meet user requirements, the visual agent module initiates the SAM+CLIP zero-shot image segmentation process, completing zero-shot segmentation through four levels of units: Image preprocessing unit: Performs standardized preprocessing on the input two-dimensional plant image. First, the image resolution is uniformly adjusted to 512×512. Then, the Z-score normalization method is used to normalize the pixel values ​​of the RGB three channels of the image to eliminate the interference caused by differences in light intensity and shooting angle, while preserving the key phenotypic features of the plant. Candidate Mask Generation Unit: The preprocessed plant image is input into the SAM pre-trained model. Through the model's built-in sliding window anchor point generation mechanism, a full-image-level candidate mask set is generated at different scales of the image. The number of candidate masks is controlled to be 150-200. Each mask is represented by a binary matrix, and the mask resolution is consistent with the input image. At the same time, a non-maximum suppression algorithm is used to initially filter redundant masks with an overlap of ≥70%, and finally a candidate mask set of 45-60 is obtained. Feature matching unit: The plant name obtained from the above image classification steps is used as a prompt word and input into the text encoder of the CLIP model to generate a text feature vector of the plant category; at the same time, each candidate mask generated by SAM is input into the image encoder of the CLIP model to extract the image feature vector of each mask region; by calculating the cosine similarity between the text feature vector and the feature vector of each mask image, the matching score between each candidate mask and the target plant category is obtained, and the matching score is normalized to the [0,1] interval. The higher the score, the higher the confidence that the area corresponding to the mask is the target plant. Mask fusion unit: The Top-K adaptive selection algorithm is adopted, and the K value is the rounded result of the total number of candidate masks × 0.3. The high confidence mask regions with the top K matching scores are selected. All high confidence masks are fused at the pixel level to obtain the complete plant target segmentation mask, thus completing the zero-shot plant image segmentation task.

[0039] (3) Implementation of target detection, 3D reconstruction and point cloud segmentation tasks Target detection task: To meet the needs of plant count detection, organ identification and counting, etc., the YOLO model is called to perform target detection processing on the input plant image to complete the recognition and counting tasks; 3D Reconstruction Task: Based on the user-input 2D plant multi-view image data, the built-in VGGT model is called to calculate and generate a 3D point cloud model of the plant end-to-end, thus completing the 3D reconstruction task of the plant point cloud. Point cloud segmentation task: Based on the 3D plant point cloud data input by the user, the built-in PointNet++ model is invoked to complete the point cloud segmentation task at the plant organ level.

[0040] Figure 4 The results of three-dimensional point cloud segmentation of a single cabbage plant were shown before and after comparison.

[0041] (4) Extraction of plant phenotypic parameters Based on the above image processing results, the phenotypic parameter calculation toolbox is called to extract the corresponding plant phenotypic parameters, including but not limited to plant height, plant width, leaf length, and leaf width, to provide core phenotypic data for subsequent GWAS analysis and paper writing.

[0042] Figure 3 The document demonstrates the workflow for extracting phenotypic parameters from a single rice plant image. The dialog box on the right illustrates the interaction between the user and the multi-agent collaborative system: the user issues a task command to the system, with the core requirement being phenotypic parameter extraction, and the input data being a single rice plant image; after receiving the command, the planning agent module first parses and breaks down the user's task, then schedules the visual agent module to execute the phenotypic parameter extraction task. This module uses the image segmentation results to extract the rice phenotypic parameters and displays the extracted phenotypic parameters in a table at the bottom center of the interface; finally, the planning agent module summarizes the entire task execution process and results, and provides the user with the final conclusion.

[0043] 3. Model Fine-tuning Agent Module In this embodiment, when the model fine-tuning agent module is scheduled and executed by the planning agent module, it fine-tunes and optimizes the neural network model in the visual agent module based on the user-provided labeled structured plant dataset. The specific implementation method is as follows: (1) Hyperparameter optimization First, the hyperparameter optimization unit is started, and the Bayesian hyperparameter optimization algorithm is used to perform a global search on key hyperparameters such as learning rate, weight decay, batch size, and training epochs. Through multiple rounds of iterative verification, the optimal training configuration that fits the current dataset and the target model is determined, providing parameter support for subsequent model training.

[0044] (2) Parameter fine-tuning optimization Different fine-tuning strategies are adopted for models with different numbers of parameters: Full parameter training: For neural network models with a moderate number of parameters, such as SegFormer, YOLO, and DeepLabV3, full parameter training is performed based on the user-provided labeled structured plant image dataset. All trainable parameters of the model are updated and iterated to enhance the model's adaptability to the dataset. Lightweight fine-tuning: For pre-trained models with a large number of parameters such as SAM, CLIP, and BLIP, based on the labeled structured plant image dataset provided by the user, the LoRA low-rank adaptation algorithm is used to fine-tune only the key parameters of the model's attention layer. Under the premise of controlling training costs and reducing computational resource consumption, the model's adaptability to the target dataset is effectively enhanced, and the model's phenotypic analysis performance is guaranteed.

[0045] (3) Model architecture optimization In this embodiment, the model fine-tuning agent module can also optimize the architecture of the neural network model in the visual agent module. Specifically, it adopts a teacher-student collaborative training strategy and combines a neural architecture search algorithm to automatically search and optimize the internal structure of the neural network model. Under the premise of controlling the number of model parameters and computational load, it improves the execution accuracy and inference speed of the visual agent module in plant image processing tasks.

[0046] 4. Code-based intelligent agent module In this embodiment, when the code agent module is scheduled and executed by the planning agent module, it automatically writes and executes computer code according to task requirements. The specific implementation is as follows: First, based on the task requirements and input data issued by the planning agent module, the data processing objectives, statistical analysis requirements, and chart drawing specifications are clarified through the underlying large language model. In a secure Python sandbox environment, corresponding executable Python code is automatically generated using code libraries including Pandas and Matplotlib. Second, the generated code is executed in the sandbox environment to complete data processing and customized chart drawing operations. Third, if errors occur or tasks are not completed during code execution, the introspective error tracking capability of the toolbox is used to read error information, locate syntax errors, logical errors, and dependency problems in the code, autonomously correct the code, and re-execute until the code runs successfully and completes all tasks assigned by the planning agent module.

[0047] 5. GWAS Analysis Agent Module In this embodiment, when the GWAS analysis agent module is scheduled and executed by the planning agent module, it performs genome-wide association analysis based on the genotype data provided by the user and the plant phenotypic parameters output by the visual agent module. The specific implementation is as follows: First, the input genotype data is quality controlled to remove SNPs with low detection rates, excessively low minor allele frequencies, and significant deviations from Hardy-Weinberg equilibrium. Simultaneously, the samples are filtered for phylogenetic relationships and population stratification. Second, using the plant phenotypic parameters output by the visual agent module as phenotypic data and the quality-controlled genotype data as genotype input, population structure analysis and phylogenetic matrix calculation are performed to control for false positive associations caused by population stratification. Third, a mixed linear model is used to perform genome-wide association analysis, calculating the significance p-value of the association between each SNP locus and the target phenotypic trait. Fourth, based on the association analysis results, Manhattan plots and QQ plots are drawn for visualization, while significantly associated SNP loci are screened to identify candidate genes related to the target phenotypic trait.

[0048] 6. Thesis Writing Intelligent Agent Module In this embodiment, when the paper writing agent module is scheduled and executed by the planning agent module, it automatically completes the writing and optimization of scientific research papers based on the multimodal data provided by the user and the processing results output by the multi-agent collaborative system. This is specifically implemented through a three-level unit: Literature retrieval and knowledge base construction unit: Activate the Internet deep search function, and retrieve high-level academic literature and review articles in relevant fields from Google Scholar based on the research direction and topic suggested by the user. Use search enhancement generation technology to convert the retrieved literature content into vector data and store it in the vector knowledge base of the writing agent to provide literature support for paper writing and ensure the logic, accuracy and academic norms of the paper's introduction and discussion section. Paper writing unit: Based on the full-process processing results output by the multi-agent collaborative system, the scientific research paper is automatically written in accordance with the standard structure of academic papers. Among them, the plant phenotypic parameters obtained by the visual agent module and the correlation analysis results obtained by the GWAS analysis agent module are used as the core data of the paper results and analysis section. Customized charts generated by the code agent module are used as the visualization content of the paper, ensuring the integrity, scientificity and standardization of the paper content. Paper Review and Optimization Unit: After the initial draft of the paper is completed, the paper content is reviewed and iterated multiple times, including grammatical correction, terminology standardization, logical streamlining, structural optimization, format adjustment, and reference verification. This corrects errors in expression and logical inaccuracies in the paper, continuously improves the paper quality, and ultimately completes the writing of an academic paper that meets scientific research standards.

[0049] The specific examples described in this application are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the specific examples described herein, or substitute them by similar means, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A plant phenotypic analysis intelligent agent system based on a large language model, characterized in that, This includes a multi-agent collaborative system. The system uses a large language model as its core inference engine and integrates six dedicated agent modules: a planning agent module, a visual agent module, a model fine-tuning agent module, a code agent module, a GWAS analysis agent module, and a paper writing agent module. Each of these six dedicated agent modules is equipped with a corresponding toolbox and neural network model to perform its respective phenotypic analysis-related tasks. The planning agent module is used to parse and analyze user needs based on user input prompts and multimodal data, plan the overall task execution process based on the analysis results, break down the overall task into multiple sub-tasks, and assign each sub-task to a corresponding dedicated agent module for sequential execution; it is also used to summarize and integrate the execution results of the entire agent system after all planned sub-tasks have been executed by their respective dedicated agent modules, and feed back the processed final result to the user. The visual intelligent agent module is communicatively connected to the planning intelligent agent module. When scheduled and executed by the planning intelligent agent module, it performs image classification, image segmentation, and target detection processing operations based on the two-dimensional plant image data input by the user; performs three-dimensional reconstruction processing operations based on the two-dimensional plant multi-view image data input by the user; performs point cloud segmentation processing operations based on the three-dimensional plant point cloud data input by the user; and extracts the corresponding plant phenotypic parameters based on the above processing results. The model fine-tuning agent module is communicatively connected to the planning agent module. When scheduled and executed by the planning agent module, it is used to fine-tune and optimize the neural network model in the visual agent module based on the labeled structured plant dataset provided by the user, adjusting the parameters or model architecture of the neural network model to make the neural network model more suitable for the plant dataset provided by the user. The code intelligence module is communicatively connected to the planning intelligence module. When scheduled for execution by the planning intelligence module, it automatically writes and executes computer code according to task requirements to complete data processing and customized chart drawing operations. If an error occurs or the task is not completed during execution, the computer code is corrected according to the error information and re-executed until the task assigned by the planning intelligence module is completed. The GWAS analysis agent module is communicatively connected to the planning agent module. When scheduled and executed by the planning agent module, it performs population structure analysis and kinship matrix calculation based on the genotype data provided by the user and the plant phenotypic parameters output by the visual agent module, and finally draws Manhattan diagrams and QQ diagrams to discover significantly associated SNP loci. The paper writing intelligent agent module communicates with the planning intelligent agent module. When scheduled and executed by the planning intelligent agent module, it automatically generates and writes scientific research papers based on the multimodal data provided by the user and the processing results output by the multi-agent collaborative system. After the first draft of the paper is completed, the paper content is autonomously reviewed and iteratively optimized.

2. The intelligent agent system for plant phenotyping based on a large language model according to claim 1, characterized in that, The multi-agent collaborative system adopts a star topology architecture, with the planning agent module as the core scheduling node of the system, and the other dedicated agent modules as execution nodes connected to the core scheduling node. The planning agent module completes task allocation, instruction forwarding and result aggregation. Each intelligent agent module uses the Qwen-2.5 large language model as a unified underlying reasoning engine, relying on the large language model to realize intent understanding, logical reasoning, decision planning and content generation; The multi-agent collaborative system also has a unified and standardized application interface for interactive operations such as issuing task instructions, transmitting execution status, transmitting intermediate data, and reporting processing results. Each dedicated intelligent agent module is independently configured with a toolbox and neural network model that matches its function, in order to carry out specialized processing capabilities and realize specialized division of labor and collaborative operation under a unified scheduling framework.

3. The intelligent agent system for plant phenotyping based on a large language model according to claim 1, characterized in that, The planning agent module, as the core scheduling node of the system, specifically executes subtask allocation and scheduling according to the following rules: When the user prompt contains image processing information such as image segmentation, image classification, and phenotypic parameter extraction, and the input data is a single plant image, the planning agent module schedules the visual agent module to execute the plant image processing task. When the user prompt contains 3D reconstruction information and the input data is a multi-view image of a plant, the planning agent module schedules the visual agent module to perform the 3D reconstruction task of the plant point cloud. When the user prompt contains point cloud segmentation information and the input data is a three-dimensional point cloud of a plant, the planning agent module schedules the visual agent module to perform the three-dimensional point cloud segmentation task of the plant. When the user prompt contains optimization model information and the input data is a labeled structured plant dataset, the planning agent module schedules the model fine-tuning agent module to execute the neural network model optimization task; When the user prompt includes information on drawing charts and cleaning data, and the input data is in Excel and CSV format, the planning agent module schedules the code agent module to automatically write computer code and execute it to complete the task. When the user prompt contains gene analysis information and the input data is genotype data, the planning agent module schedules the GWAS analysis agent module to perform the task of calculating the relationship between genotype and phenotype, and draws a Manhattan diagram and a QQ diagram to visualize the results. When the user prompt includes information about writing a paper, the planning agent module schedules the paper writing agent module to execute the paper writing task.

4. The intelligent agent system for plant phenotyping based on a large language model according to claim 1, characterized in that, When the visual intelligent agent module performs plant image classification, it is specifically implemented through the following units: The feature extraction unit is used to extract the feature vector f of the plant image using the DINOV3 pre-trained feature extractor in the toolbox. p Meanwhile, the visual intelligent agent module pre-sets feature vectors for various plants. m represents the number of plant feature vectors contained in the vector; Similarity matching unit, used to match feature vector f p With m eigenvectors f i Cosine similarity is calculated for each value individually, resulting in m cosine similarity values. If C i The maximum value C in max If the value is greater than the preset threshold τ, then the matching between the user-input plant image and the preset plant image is completed, thus completing the image classification task; if C i The maximum value C in max If the value is less than the preset threshold τ, the language description classification unit is triggered; The language description classification unit integrates a BLIP pre-trained neural network model. When the similarity matching unit cannot complete the plant image classification task, the BLIP model describes the plant image input by the user in human natural language. Then, the big language model inference core performs named entity recognition on the natural language description to obtain the plant name in the natural language description and complete the plant image classification.

5. The intelligent agent system for plant phenotyping based on a large language model according to claim 1, characterized in that, The visual intelligent agent module integrates three neural network models to handle plant image processing tasks at different scales. The SegFormer model is used for plant cell microscopic image segmentation, the YOLO model is used for organ-level segmentation and detection counting of single plant images, and the DeepLabV3 model is used for field plant image segmentation by UAVs.

6. The intelligent agent system for plant phenotyping based on a large language model according to claim 5, characterized in that, The visual intelligent agent module also integrates SAM and CLIP pre-trained neural network models, which are used to perform zero-shot image segmentation when the three neural network models cannot complete the plant image segmentation task or the segmentation results cannot meet the user's needs. This is specifically implemented through the following units: The image preprocessing unit is used to perform standardized preprocessing on the input two-dimensional plant image, uniformly adjusting the image resolution to 512×512. Then, the Z-score normalization method is used to normalize the pixel values ​​of the RGB three channels of the image to eliminate the interference caused by differences in light intensity and shooting angle, while preserving the key phenotypic features of the plant. The candidate mask generation unit is used to input the preprocessed plant image into the SAM pre-trained model. Through the model's built-in sliding window anchor point generation mechanism, it generates a full-image-level candidate mask set at different scales of the image. The number of candidate masks is controlled to be 150-200. Each mask is represented by a binary matrix. The mask resolution is consistent with the input image. At the same time, the non-maximum suppression algorithm is used to initially filter redundant masks with an overlap of ≥70%, and finally obtains a candidate mask set of 45-60. The feature matching unit is used to input the plant image classification name as described in claim 4 as a prompt word into the text encoder of the CLIP model to generate a plant category text feature vector; at the same time, it inputs each candidate mask set generated by SAM into the image encoder of the CLIP model to extract the image feature vector of the mask region; by calculating the cosine similarity between the text feature vector and each mask image feature vector, the matching score between each candidate mask and the target plant category is obtained, and the matching score is normalized to the [0,1] interval. The higher the score, the higher the confidence that the mask region is the target plant. The mask fusion unit is used to employ the Top-K adaptive selection algorithm, where the K value is equal to the rounded-up result of the total number of candidate masks × 0.3, to select the K high-confidence mask regions with the highest matching scores; all high-confidence masks are fused to obtain the complete plant target segmentation mask, thus completing the zero-shot plant image segmentation task.

7. The intelligent agent system for plant phenotyping based on a large language model according to claim 1, characterized in that, The visual intelligent agent module incorporates the VGGT model and the PointNet++ model. The VGGT model is used for 3D reconstruction of plant point clouds, directly calculating the 3D point cloud model of the plant based on the multi-view image of the plant input by the user. The PointNet++ model is used for plant point cloud segmentation, directly completing the point cloud segmentation of plant organs based on the 3D point cloud of the plant input by the user.

8. The intelligent agent system for plant phenotyping based on a large language model according to claim 1, characterized in that, The model fine-tuning agent module optimizes the parameters of the neural network model in the visual agent module, specifically through the following units: The hyperparameter optimization unit is used to perform a global search on key hyperparameters such as learning rate, weight decay and batch size using the Bayesian hyperparameter optimization algorithm to determine the optimal training configuration. The full training unit is used to perform full parameter training on the SegFormer, YOLO, and DeepLabV3 neural network models with appropriate parameter amounts in the visual agent module, based on the structured plant image dataset provided by the user, so as to enhance the adaptability of the neural network model to the dataset. The lightweight fine-tuning unit is used to fine-tune some parameters of the neural network model with a large number of SAM, CLIP and BLIP parameters in the visual agent module, based on the structured plant image dataset provided by the user, using the LoRA algorithm, so as to enhance the adaptability of the neural network model to the dataset. The model fine-tuning agent module is also used to optimize the model architecture of the neural network model in the visual agent module. Specifically, it adopts a teacher-student collaborative training strategy and combines a neural architecture search algorithm to search and optimize the internal structure of the neural network model in order to improve the execution accuracy of the visual agent module in plant image processing tasks.

9. The intelligent agent system for plant phenotyping based on a large language model according to claim 1, characterized in that, The code agent module automatically writes and executes computer code. Specifically, within a secure Python sandbox, the toolbox of the code agent module automatically generates code using libraries including Pandas and Matplotlib to complete data processing and customized chart drawing tasks. The toolbox has introspective error tracking capabilities, used to read error messages when code execution fails, correct logical errors in the code, and re-execute the computer code until all assigned tasks are successfully completed.

10. The intelligent agent system for plant phenotyping based on a large language model according to claim 1, characterized in that, The paper writing intelligent agent module specifically implements paper writing through the following units: The literature retrieval and knowledge base construction unit has a deep internet search function, which is used to search for academic literature and review articles in relevant fields from Google Scholar based on user prompts. Then, it uses retrieval enhancement generation technology to transfer the searched academic literature and review articles into the vector knowledge base of the writing agent. The paper writing unit is used to automatically complete the writing of scientific papers based on the processing results output by the multi-agent collaborative system. Among them, the plant phenotypic parameters obtained by the visual agent module and the GWAS analysis results obtained by the GWAS analysis agent module are used as the core data for paper analysis, and the customized charts generated by the code agent module are used for the visualization of the analysis results. The paper review and optimization unit is used to independently review and iteratively optimize the content of the paper after the initial draft is completed, until the scientific paper is completed.

Images