Crop phenotype analysis method and related device
By combining an image encoder and a multimodal large language model, the problem of extracting features from irregularly shaped regions in crop phenotypic analysis in existing technologies has been solved, achieving high-precision and high-reliability crop phenotypic analysis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- XI AN JIAOTONG UNIV
- Filing Date
- 2026-01-05
- Publication Date
- 2026-04-24
AI Technical Summary
Existing technologies struggle to accurately extract features from irregularly shaped regions in complex agricultural environments during crop phenotypic analysis, resulting in poor analytical results. Furthermore, existing models lack deep reasoning capabilities and interactivity, failing to provide interpretable diagnostic reports.
Visual features are extracted using a pre-trained image encoder, and local geometric topology aggregation is performed by sampling points through region masking and combined with a multimodal large language model for analysis to generate high-fidelity multimodal fusion inference results.
It achieves pixel-level feature extraction for regions of arbitrary shapes, eliminates background noise interference, significantly improves feature alignment accuracy and inference accuracy, reduces the error response rate, and enhances the credibility of analysis results.
Smart Images
Figure CN121921650A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of multimodal artificial intelligence and agricultural information technology, and relates to a method and related apparatus for crop phenotypic analysis. Background Technology
[0002] With the development of concepts such as precision agriculture, smart agriculture, and the agricultural metaverse, crop phenotyping has become a crucial link in modern breeding and agricultural production. Crop phenotypic data forms the basis for breeding decisions using artificial intelligence and genomic tools. Traditional crop phenotyping relies primarily on manual observation and field measurements, a method that is not only labor-intensive and time-consuming but also highly susceptible to subjective errors. To overcome these limitations, early computational methods attempted to automate the process by combining hand-designed features with classical artificial intelligence techniques. However, these methods are highly sensitive to environmental conditions and cannot adapt to complex field environments because hand-designed features struggle to capture the highly variable phenotypic characteristics of crops in the real world.
[0003] With the rise of deep learning technology, general-purpose visual models based on convolutional neural networks have been widely used in tasks such as crop identification, disease detection, and pest monitoring. These models, through automatic feature learning, significantly outperform classical AI methods in terms of robustness and accuracy. Furthermore, the application of large language models in agriculture has gradually increased in recent years. Despite the progress made by the aforementioned technologies, existing techniques still have some significant shortcomings and deficiencies in practical crop phenotypic analysis applications.
[0004] Specifically, traditional visual models lack deep reasoning capabilities and interactivity. Current agricultural visual models are mainly limited to providing basic classification labels or bounding box outputs. Their architecture is designed only to complete recognition tasks but lacks domain-specific reasoning capabilities. Unlike large language models, they cannot provide interpretive diagnostic reports or engage in multi-turn dialogues with users, limiting their application depth in complex agricultural decision-making. Furthermore, agricultural large language models have visual blind spots. While existing agricultural-specific large language models perform well in text processing, most are text-centric, relying on user-provided text descriptions for diagnosis or recommendations. They lack the ability to directly perceive image information. This reliance on subjective text descriptions can easily lead to incomplete or inaccurate phenotypic analysis results due to unclear or omitted descriptions. Additionally, crop phenotypic features exhibit extremely high geometric complexity and non-convex shape characteristics. Faced with this complex unstructured scene, mainstream region-aware models mainly rely on rectangular bounding boxes to locate targets. However, in agricultural scenarios, these bounding boxes inevitably contain a large amount of irrelevant background. This background noise severely dilutes the feature signals of small lesions, leading to a decrease in diagnostic accuracy. Meanwhile, although some methods employ mask-based feature extractors to handle irregular shapes, these models are primarily designed for general scenarios. When directly applied to complex, unstructured agricultural environments, they face significant generalization challenges and struggle to accurately extract fine-grained phenotypic features. For these reasons, existing methods cannot accurately analyze the phenotypic features of irregularly shaped regions such as crop lesions, resulting in unsatisfactory crop phenotypic analysis results. Summary of the Invention
[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and related apparatus for crop phenotypic analysis.
[0006] To achieve the above objectives, the present invention employs the following technical solution: In a first aspect, the present invention provides a method for crop phenotypic analysis, comprising: acquiring an original crop image and extracting original visual features of the original crop image based on a pre-trained image encoder; generating in parallel a general visual feature vector representing the global context of the image and a visual feature map preserving the spatial structure of the image based on the original visual features; acquiring a region mask of the original crop image and sampling several sampling points from the foreground of the region mask, and acquiring the visual feature vectors corresponding to the sampling points in the visual feature map; performing local geometric topological aggregation on the visual feature vectors corresponding to the sampling points in the visual feature map to obtain a region visual feature vector of the original crop image; acquiring the analysis question text, constructing a multimodal sequence by combining the general visual feature vector and the region visual feature vector, and calling a preset multimodal large language model to obtain the crop phenotypic analysis result.
[0007] Optionally, the step of acquiring the original crop image and extracting the original visual features of the original crop image based on a pre-trained image encoder, and generating a general visual feature vector representing the global context of the image and a visual feature map that preserves the spatial structure of the image in parallel based on the original visual features, includes: dividing the original crop image into several local sub-images and scaling the original crop image into a panoramic reference image; extracting the visual features of each local sub-image and the panoramic reference image based on the pre-trained image encoder; extracting and compressing the visual features of each local sub-image and the panoramic reference image through a pre-trained resampler based on a cross-attention mechanism to obtain extracted features and projecting them into the embedding space of a preset multimodal large language model to obtain a general visual feature vector representing the global context of the image; stitching the visual features of each local sub-image according to the spatial position of each local sub-image in the original crop image to obtain a high-resolution feature map; and performing bilinear interpolation upsampling on the visual features of the panoramic reference image to make its spatial size match the high-resolution feature map. Figure 1 The high-resolution feature map is then fused with the panoramic reference feature map element by element to obtain the visual feature map of the original crop image.
[0008] Optionally, the step of obtaining the region mask of the original crop image and sampling several sampling points from the foreground of the region mask, and obtaining the visual feature vectors corresponding to the sampling points in the visual feature map, includes: obtaining the region mask of the original crop image and randomly sampling several sampling points from the foreground of the region mask, and normalizing the coordinates of each sampling point to obtain the normalized coordinates of each sampling point; wherein, the region mask of the original crop image is obtained by freely selecting on the visualized original crop image; based on the normalized coordinates of each sampling point, the feature vectors corresponding to the normalized coordinates of each sampling point are extracted in the visual feature map by bilinear interpolation.
[0009] Optionally, the step of performing local geometric topological aggregation of the visual feature vectors corresponding to several sampling points in the visual feature map to obtain the regional visual feature vector of the original crop image includes: sampling several backbone points from several sampling points using the farthest point sampling method; constructing a local geometric neighborhood for each backbone point based on several sampling points using the K-nearest neighbor algorithm, and obtaining the feature difference vector between the feature vectors of each neighbor point in the local geometric neighborhood of each backbone point and the feature vector of each backbone point, to obtain several feature difference vectors of each backbone point; extracting the several feature difference vectors of each backbone point through a pre-trained fully connected layer to obtain the local gradient feature vector of each backbone point; concatenating the local gradient feature vectors and the visual feature vectors of each backbone point, and aggregating them through a max pooling layer to obtain the enhanced local feature vector of each backbone point; flattening the enhanced local feature vectors of all backbone points, and mapping them to a global feature vector through a pre-trained first linear projection layer; projecting the global feature vector to a dimension consistent with the hidden layer of the preset multimodal large language model through a pre-trained second linear projection layer to obtain the regional visual feature vector of the original crop image.
[0010] Optionally, the step of obtaining the analysis question text, constructing a multimodal sequence by combining general visual feature vectors and regional visual feature vectors, and calling a preset multimodal large language model to obtain crop phenotypic analysis results includes: encoding the analysis question text into text vectors, embedding the text vectors and regional visual feature vectors into preset multimodal sequence templates respectively to obtain a text feature sequence fused with regional semantics; concatenating the general visual feature vectors with the text feature sequence fused with regional semantics to obtain a multimodal input sequence; inputting the multimodal sequence into the preset multimodal large language model to obtain crop phenotypic analysis results; wherein, the preset multimodal large language model is obtained by training an improved multimodal large language model using a crop phenotypic analysis dataset; wherein, the improved multimodal large language model is obtained by replacing the last layer of the decoder of the multimodal large language model with a sparse hybrid expert layer containing several expert networks; wherein, the several expert networks of the sparse hybrid expert layer calculate the activation probability of each expert network according to the multimodal sequence through a preset routing network, and control the participation of a preset number of expert networks with high activation probabilities in the calculation through a Top-K gating mechanism based on the activation probability of each expert network.
[0011] Optionally, when training the improved multimodal large language model using the crop phenotypic analysis dataset, the resampler, fully connected layer, first linear projection layer, and second linear projection layer based on the cross-attention mechanism are fine-tuned for all parameters; the decoder of the improved multimodal large language model is fine-tuned for parameters using a low-rank adaptive method; and the training loss function is an autoregressive language modeling loss function with added load balancing loss terms from several expert networks.
[0012] Optionally, the crop phenotypic analysis dataset is constructed as follows: Several crop phenotypic analysis training image samples and key phenotypic expert annotation results for each sample are obtained; the key phenotypic expert annotation results for each training image sample are used as semantic anchors and embedded together with preset analysis question text samples into a preset large language model structured template to obtain strong constraint input prompts for each training image sample; each training image sample and its strong constraint input prompts are input into a multimodal large language model to obtain crop phenotypic analysis results for each training image sample; the training image samples, analysis question text samples, and crop phenotypic analysis results are integrated to form a crop phenotypic analysis dataset containing several sample pairs; wherein, a sample pair is a pair of crop phenotypic analysis training image samples, analysis question text samples, and crop phenotypic analysis results of crop phenotypic analysis training image samples.
[0013] In a second aspect, the present invention provides a crop phenotypic analysis system, comprising: an extraction module for acquiring an original crop image and extracting original visual features of the original crop image based on a pre-trained image encoder; generating in parallel a general visual feature vector representing the global context of the image and a visual feature map preserving the spatial structure of the image based on the original visual features; a sampling module for acquiring a region mask of the original crop image and sampling several sampling points from the foreground of the region mask, and acquiring the visual feature vectors corresponding to the sampling points in the visual feature map; an aggregation module for performing local geometric topological aggregation of the visual feature vectors corresponding to the sampling points in the visual feature map to obtain a region visual feature vector of the original crop image; and an analysis module for acquiring analysis question text, constructing a multimodal sequence by combining the general visual feature vector and the region visual feature vector, and calling a preset multimodal large language model to obtain crop phenotypic analysis results.
[0014] In a third aspect, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described crop phenotypic analysis method.
[0015] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described crop phenotypic analysis method.
[0016] Compared with the prior art, the present invention has the following beneficial effects: This invention provides a crop phenotypic analysis method. First, it acquires the original crop image and extracts its original visual features based on a pre-trained image encoder. Then, based on these original visual features, it generates a general visual feature vector representing the global context of the image and a visual feature map that preserves the image's spatial structure. Next, it acquires a region mask of the original crop image and samples several sampling points from the foreground of the region mask, as well as the corresponding visual feature vectors of these sampling points in the visual feature map. Finally, it performs local geometric topological aggregation on the visual feature vectors corresponding to these sampling points in the visual feature map to obtain the region visual feature vector of the original crop image. This method enables pixel-level feature extraction of regions of arbitrary shapes. This design effectively eliminates interference from irrelevant background noise, significantly improving the feature alignment accuracy and inference accuracy of the model in fine-grained phenotypic analysis tasks (such as the detection of small lesions). It effectively solves the problem that existing technologies often rely on rectangular bounding boxes to locate targets, making it difficult to handle highly irregular phenotypic features such as crop lesions and leaf necrosis. Meanwhile, when calling the preset multimodal large language model for analysis, the extracted high-dimensional regional visual features are explicitly embedded into the text sequence as visual anchors, ensuring that the model's analysis is strictly based on real visual evidence, achieving high-fidelity multimodal fusion reasoning, significantly reducing the proportion of erroneous responses, and improving the credibility of the analysis results. Attached Figure Description
[0017] Figure 1 This is a flowchart of the crop phenotypic analysis method according to an embodiment of the present invention.
[0018] Figure 2 This is a schematic diagram of the visual interactive terminal interface according to an embodiment of the present invention.
[0019] Figure 3 This is a flowchart detailing the feature extraction and phenotypic analysis in an embodiment of the present invention.
[0020] Figure 4 This is a schematic diagram illustrating the principle of constructing a crop phenotypic analysis dataset according to an embodiment of the present invention.
[0021] Figure 5 This is a system structure block diagram of the crop phenotypic analysis method according to an embodiment of the present invention. Detailed Implementation
[0022] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0024] The present invention will now be described in further detail with reference to the accompanying drawings: See Figure 1 In one embodiment of the present invention, a crop phenotypic analysis method is provided, which can achieve efficient, accurate and reliable crop phenotypic analysis in complex agricultural scenarios.
[0025] Specifically, the crop phenotypic analysis method of the present invention includes the following steps: S1: Obtain the original crop image and extract the original visual features of the original crop image based on the pre-trained image encoder; generate in parallel a general visual feature vector representing the global context of the image and a visual feature map that preserves the spatial structure of the image based on the original visual features.
[0026] S2: Obtain the region mask of the original crop image and sample several sampling points from the foreground of the region mask, and obtain the visual feature vectors corresponding to the sampling points in the visual feature map.
[0027] S3: Perform local geometric topological aggregation on the visual feature vectors corresponding to several sampling points in the visual feature map to obtain the regional visual feature vectors of the original crop image.
[0028] S4: Obtain the text of the analysis question, and construct a multimodal sequence by combining general visual feature vectors and regional visual feature vectors, and call the preset multimodal large language model to obtain the crop phenotypic analysis results.
[0029] This invention provides a crop phenotypic analysis method. First, it acquires the original crop image and extracts its original visual features based on a pre-trained image encoder. Then, based on these original visual features, it generates a general visual feature vector representing the global context of the image and a visual feature map that preserves the image's spatial structure. Next, it acquires a region mask of the original crop image and samples several sampling points from the foreground of the region mask, as well as the corresponding visual feature vectors of these sampling points in the visual feature map. Finally, it performs local geometric topological aggregation on the visual feature vectors corresponding to these sampling points in the visual feature map to obtain the region visual feature vector of the original crop image. This method enables pixel-level feature extraction of regions of arbitrary shapes. This design effectively eliminates interference from irrelevant background noise, significantly improving the feature alignment accuracy and inference accuracy of the model in fine-grained phenotypic analysis tasks (such as the detection of small lesions). It effectively solves the problem that existing technologies often rely on rectangular bounding boxes to locate targets, making it difficult to handle highly irregular phenotypic features such as crop lesions and leaf necrosis. Meanwhile, when calling the preset multimodal large language model for analysis, the extracted high-dimensional regional visual features are explicitly embedded into the text sequence as visual anchors, ensuring that the model's analysis is strictly based on real visual evidence, achieving high-fidelity multimodal fusion reasoning, significantly reducing the proportion of erroneous responses, and improving the credibility of the analysis results.
[0030] In one possible implementation, the steps of acquiring the original crop image and extracting the original visual features of the original crop image based on a pre-trained image encoder, and generating a general visual feature vector representing the global context of the image and a visual feature map preserving the spatial structure of the image in parallel based on the original visual features, include: dividing the original crop image into several local sub-images and scaling the original crop image into a panoramic reference image; extracting visual features of each local sub-image and the panoramic reference image based on the pre-trained image encoder; extracting and compressing the visual features of each local sub-image and the panoramic reference image through a pre-trained resampler based on a cross-attention mechanism to obtain extracted features and projecting them into the embedding space of a preset multimodal large language model to obtain a general visual feature vector representing the global context of the image; stitching the visual features of each local sub-image according to the spatial position of each local sub-image in the original crop image to obtain a high-resolution feature map; and performing bilinear interpolation upsampling on the visual features of the panoramic reference image to make its spatial size match the high-resolution feature map. Figure 1 The high-resolution feature map is then fused with the panoramic reference feature map element by element to obtain the visual feature map of the original crop image.
[0031] Explanatoryly, this step aims to obtain high-resolution visual features that preserve local details.
[0032] First, the original crop image is received, and an adaptive slicing strategy is used to divide it into segments based on the resolution and aspect ratio of the original crop image. The grid, and then generate A local sub-image is generated, while a scaled panoramic reference image is retained to maintain the global context. Then, a pre-trained image encoder is used to segment all local sub-images and the panoramic reference image into patches for parallel encoding, thereby outputting multi-level visual features. Simultaneously, to obtain the image features for extraction using the mask, the visual features of each local sub-image are spatially stitched and recombined according to the grid positions to restore the spatial structure of the image. These features are then fused element-wise with the visual features of the panoramic reference image after bilinear interpolation upsampling to generate a unified visual feature map of the original crop image. ;in, For the reorganized spatial dimensions, For feature dimensions.
[0033] Furthermore, to enable the multimodal large language model to understand the overall semantic environment of an image, a resampler based on a cross-attention mechanism is used, leveraging a set of learnable query vectors to analyze multi-level visual features. Compression is performed. This process filters out redundant visual information and projects and aligns visual features onto the semantic space of a large language model, generating a set of general visual feature vectors that represent the global context of the image.
[0034] For example, considering the need to identify small lesions in agricultural images, a pre-trained image encoder can use a SigLIP visual encoder (SigLIP-SoViT-400m / 14) to extract visual features from each local sub-image and the panoramic reference image to preserve high-resolution details.
[0035] In one possible implementation, the steps of obtaining the region mask of the original crop image and sampling several sampling points from the foreground of the region mask, and obtaining the visual feature vectors corresponding to the sampling points in the visual feature map, include: obtaining the region mask of the original crop image and randomly sampling several sampling points from the foreground of the region mask, and normalizing the coordinates of each sampling point to obtain the normalized coordinates of each sampling point; wherein, the region mask of the original crop image is obtained by freely selecting on the visualized original crop image; and based on the normalized coordinates of each sampling point, the feature vectors corresponding to the normalized coordinates of each sampling point are extracted in the visual feature map by bilinear interpolation.
[0036] For example, this embodiment provides a flexible human-computer interaction mode for region masking of raw crop images to lower the barrier to entry for non-professional users. Specifically, it supports users to specify regions of interest through free drawing and to conduct open-ended natural language question answering. This interaction method breaks the limitation of traditional models that can only handle predefined categories or fixed regions, enabling farmers or technicians to conduct targeted interactive diagnosis of specific field anomalies, which has extremely high practical application value.
[0037] For example, a user can freely draw an arbitrary shape of region of interest on a pre-developed interface displaying the original crop image. This region of interest is then processed by the core component GeoRegionSampler to obtain the region mask of the original crop image. See also Figure 2 This demonstrates a feasible visual interactive terminal interface. After loading the original crop image on the left side of the interface, the user can use a drawing tool to freely draw irregular areas such as curled leaves or scattered seeds. The interface will generate a region mask of the original crop image in real time and trigger the aforementioned process. The results of crop phenotypic analysis can be visually displayed in the dialog box on the right.
[0038] In one possible implementation, the step of performing local geometric topological aggregation of the visual feature vectors corresponding to several sampling points in the visual feature map to obtain the regional visual feature vector of the original crop image includes: sampling several backbone points from several sampling points using the farthest point sampling method; constructing a local geometric neighborhood for each backbone point based on several sampling points using the K-nearest neighbor algorithm, and obtaining the feature difference vector between the feature vectors of each neighbor point in the local geometric neighborhood of each backbone point and the feature vector of each backbone point, thereby obtaining several feature difference vectors for each backbone point; extracting the several feature difference vectors of each backbone point through a pre-trained fully connected layer to obtain the local gradient feature vector of each backbone point; concatenating the local gradient feature vectors and the visual feature vectors of each backbone point, and aggregating them through a max pooling layer to obtain the enhanced local feature vector of each backbone point; flattening the enhanced local feature vectors of all backbone points, and mapping them to a global feature vector through a pre-trained first linear projection layer; and projecting the global feature vector to a dimension consistent with the hidden layer of a preset multimodal large language model through a pre-trained second linear projection layer to obtain the regional visual feature vector of the original crop image.
[0039] Explaining this, this step aims to solve the challenge of feature extraction from irregularly shaped regions such as crop lesions and cracks. To achieve accurate fitting of irregular shapes, a pixel-level point cloud processing paradigm is innovatively introduced. Through a cascaded processing flow of "sampling, interpolation, aggregation, and projection," continuous image features are transformed into semantic vectors with geometric topological structures, achieving pixel-level accurate perception of irregular agricultural targets.
[0040] Specifically, see Figure 3 Unlike traditional rectangular box pooling, this invention first directly obtains the region mask of the original crop image, and then uses a random sampling algorithm to select P0 initial points (e.g., 512) within the non-zero region, and normalizes the coordinates. This process discretizes the irregular continuous region into a pixel-level point cloud, thereby accurately fitting the geometric edge of the target and completely eliminating background pixels.
[0041] Then, based on the normalized coordinates of the above sampling points, the visual feature map of the generated original crop image is... In this study, the feature vector F0 corresponding to each normalized coordinate is extracted by bilinear interpolation algorithm, realizing the mapping of visual features from grid space to discrete point space.
[0042] Next, to capture the microscopic texture changes within the lesions, a progressive feature aggregation strategy was adopted. First, farthest point sampling (FPS) was used to filter out... There are 10 backbone points, and a local geometric neighborhood is constructed for each backbone point using the K-nearest neighbor algorithm; then the feature difference between the neighborhood points and the center point is calculated. Then, the local gradient feature vectors of each backbone point are extracted through pre-trained fully connected layers and ReLU activation functions. This enables the aggregation of micro-texture and edge gradient information around the backbone points.
[0043] Finally, the local gradient feature vectors and visual feature vectors of each backbone point are concatenated and aggregated through a max pooling layer to form enhanced local feature vectors for each backbone point. Then, the enhanced local feature vectors of all backbone points are flattened and mapped to global feature vectors through a pre-trained first linear projection layer. A pre-trained second linear projection layer is then used for dimensional transformation, ultimately compressing the visual information of irregular regions to a dimension consistent with the hidden layer of the pre-defined multimodal large language model, thus obtaining the region visual feature vectors of the original crop image. , serving as a visual anchor for subsequent reasoning.
[0044] In one possible implementation, the process of acquiring the analysis question text, constructing a multimodal sequence by combining general visual feature vectors and regional visual feature vectors, and calling a preset multimodal large language model to obtain crop phenotypic analysis results includes: encoding the analysis question text into text vectors, embedding the text vectors and regional visual feature vectors into preset multimodal sequence templates respectively to obtain a text feature sequence that integrates regional semantics; concatenating the general visual feature vectors with the text feature sequence that integrates regional semantics to obtain a multimodal input sequence; and inputting the multimodal sequence into the preset multimodal large language model to obtain crop phenotypic analysis results.
[0045] The preset multimodal large language model is obtained by training an improved multimodal large language model using a crop phenotypic analysis dataset. The improved multimodal large language model is obtained by replacing the last layer of the decoder of the multimodal large language model with a sparse hybrid expert layer containing several expert networks. The several expert networks of the sparse hybrid expert layer calculate the activation probability of each expert network according to the multimodal sequence through a preset routing network, and control the participation of a preset number of expert networks with high activation probabilities in the calculation through a Top-K gating mechanism based on the activation probability of each expert network.
[0046] Explanatory, see again Figure 3 This step achieves deep fusion and interference-resistant decoding of multimodal features. First, a pre-defined multimodal sequence template is constructed, such as an input text sequence containing special placeholders: <region> <unk>< / unk> < / region> ...in...", where <unk>These are the feature bits to be replaced. After the problem text is encoded by a tokenizer to generate a token embedding sequence, the region visual feature vector of the original crop image obtained above is directly used. Replace to <unk>The position of the vector is used to directly participate in self-attention computation as a visual semantic anchor. Subsequently, the fused sequence is fed into the decoder of a pre-defined multimodal large language model. In the last layer of the decoder (where gradient conflicts are most intense), the standard feedforward neural network (FFN) is removed and replaced with a new one containing... A sparse hybrid expert (MoE) layer of an expert network. Wherein, the expert network... The specific structure replicates the standard feedforward neural network (FFN) structure in the decoder layer of the multimodal large language model; specifically, each expert network consists of a gated linear unit (GLU) or a multilayer perceptron containing an upper projection layer, a nonlinear activation function, and a lower projection layer.
[0047] For inputting a Token Routing network The activation probability of each expert network is calculated by analyzing the task attributes of the input features. And only the highest probability activation is performed through the Top-K gating mechanism. Several experts participated in the calculation, and the calculation formula is as follows: Ultimately, it outputs accurate crop phenotypic analysis results. This mechanism achieves computational decoupling of different agricultural tasks (such as disease detection and freshness assessment) through dynamic routing, effectively solving the interference between multiple tasks such as disease diagnosis, counting, and assessment.
[0048] Based on the above design, this invention effectively solves the problems of gradient conflict and negative transfer in multi-task learning. Specifically, when dealing with heterogeneous agricultural tasks (such as semantic classification and spatial coordinate generation), traditional shared-parameter models often suffer performance degradation due to differences in optimization objectives. This invention innovatively designs a sparse activation task-adaptive routing mechanism at the decoder end, which can dynamically activate specific expert networks based on the semantic features of the input content. This mechanism maintains the sharing of common features at the lower level while achieving decoupling and specialization of high-level task characteristics, effectively avoiding gradient interference between multiple tasks, and significantly improving the overall performance of the model in various tasks such as disease detection, maturity assessment, and counting.
[0049] In one possible implementation, when training the improved multimodal large language model using a crop phenotypic analysis dataset, full parameter fine-tuning is performed on the resampler, fully connected layer, first linear projection layer, and second linear projection layer based on the cross-attention mechanism; the decoder of the improved multimodal large language model is fine-tuned using a low-rank adaptive method; and the training loss function is an autoregressive language modeling loss function with added load balancing loss terms from several expert networks.
[0050] Explained in this embodiment, an end-to-end training strategy with efficient parameter fine-tuning is employed during training. The parameters of the pre-trained image encoder are frozen during training to preserve general visual capabilities; full parameter fine-tuning is performed on the fully connected layers, the first linear projection layer, and the second linear projection layer to adapt to agricultural features; and a low-rank adaptive approach (LoRA) is used for the decoder of the improved multimodal large language model. Furthermore, the loss function can be a conventional autoregressive language modeling loss function, with the improvement being the introduction of several expert network load balancing loss terms into the loss function. This is to prevent routing collapse and ensure that different expert networks can autonomously specialize for specific agricultural task characteristics.
[0051] Explanatoryly, this implementation employs a parameter-efficient fine-tuning strategy for end-to-end training: pre-trained image encoder parameters are frozen to preserve general visual capabilities; full parameter fine-tuning is performed on the cross-attention-based resampler, fully connected layers, first linear projection layer, and second linear projection layer to adapt to agricultural features; and LoRA low-rank adaptive fine-tuning is used on the decoder and its sparse hybrid expert layers at the end. The loss function introduces an expert load balancing loss term based on the autoregressive language modeling loss. To prevent route collapse, training was conducted using two NVIDIA A800 GPUs, with DeepSpeedZeRO-2 and FP16 mixed-precision technology optimized for memory, and the AdamW optimizer employed. (weight decay of 0.1), in The model convergence is ensured by using an initial learning rate, a 1% warm-up, and a cosine annealing strategy.
[0052] Based on the above design, this invention significantly reduces computing resource consumption and has excellent potential for edge deployment. Compared with full parameter fine-tuning or traditional intensive hybrid expert models, this invention adopts a parameter-efficient fine-tuning strategy. This design significantly reduces memory usage (saving approximately 30GB of memory) and computational latency while ensuring high performance, solving the bottleneck of large models being difficult to run efficiently on agricultural field equipment with limited computing resources.
[0053] In one possible implementation, the crop phenotypic analysis dataset is constructed as follows: Several crop phenotypic analysis training image samples and key phenotypic expert annotation results for each sample are obtained; the key phenotypic expert annotation results for each training image sample are used as semantic anchors and embedded together with preset analysis question text samples into a preset large language model structured template to obtain strong constraint input prompts for each training image sample; the training image samples and their strong constraint input prompts are input into a multimodal large language model to obtain crop phenotypic analysis results for each training image sample; the training image samples, analysis question text samples, and crop phenotypic analysis results are integrated to form a crop phenotypic analysis dataset containing several sample pairs; wherein, a sample pair is a pair of crop phenotypic analysis training image samples, analysis question text samples, and crop phenotypic analysis results of the training image samples.
[0054] Explanatory, see Figure 4 To address the scarcity of training data for general multimodal large language models in the agricultural field, this step constructs a high-quality instruction fine-tuning dataset based on semantic anchors before model training. Specifically, firstly, training image samples containing various crop phenotypic analysis data (e.g., fruits, vegetables, and grains) are collected, and ground-truth annotations by agricultural experts on key phenotypes (e.g., disease names, symptom characteristics, or maturity levels) in these training image samples are obtained. Subsequently, these expert annotations are used as semantic anchors, embedded along with predefined analysis question text samples into a predefined structured Prompt template. This constructs a strongly semantically constrained input sequence to limit the generation boundaries of the large language model (e.g., GPT-4o). Based on the crop phenotypic analysis training image samples and semantic anchors, the large language model generates question-and-answer pairs containing multi-turn dialogues and reasoning analysis. This constructs a specialized dataset that not only possesses rich linguistic descriptions but also strictly aligns with expert facts in key diagnostic conclusions. Semantic bias is eliminated from the data source, ensuring the logical and factual accuracy of the generated visual question-and-answer pairs, providing a knowledge base free of illusions for model training.
[0055] Based on the above design, this invention establishes a dual semantic anchoring mechanism, significantly enhancing the reliability and anti-illusion capability of the generated results. Addressing the issue of multimodal large models easily generating factual illusions in specialized fields, this invention utilizes expert annotations as semantic anchors to constrain the generation of training data during the data construction phase; during the inference phase, extracted high-dimensional region visual features are explicitly embedded into the text sequence as visual anchors. This dual constraint ensures that the model's diagnostic reports and recommendations are strictly based on authentic visual evidence and expert knowledge, significantly reducing the proportion of erroneous responses and improving the credibility of agricultural intelligent diagnosis.
[0056] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.
[0057] See Figure 5 In another embodiment of the present invention, a crop phenotypic analysis system is provided, which can be used to implement the above-mentioned crop phenotypic analysis method. Specifically, the crop phenotypic analysis system includes an extraction module, a sampling module, an aggregation module, and an analysis module.
[0058] The extraction module is used to acquire the original crop image and extract the original visual features of the original crop image based on a pre-trained image encoder; based on the original visual features, a general visual feature vector representing the global context of the image and a visual feature map that preserves the spatial structure of the image are generated in parallel; the sampling module is used to acquire the region mask of the original crop image and sample several sampling points from the foreground of the region mask, and acquire the visual feature vectors corresponding to the sampling points in the visual feature map; the aggregation module is used to perform local geometric topological aggregation of the visual feature vectors corresponding to the sampling points in the visual feature map to obtain the region visual feature vector of the original crop image; the analysis module is used to acquire the analysis question text, and construct a multimodal sequence by combining the general visual feature vector and the region visual feature vector, and call a preset multimodal large language model to obtain the crop phenotypic analysis results.
[0059] In one possible implementation, the steps of acquiring the original crop image and extracting the original visual features of the original crop image based on a pre-trained image encoder, and generating a general visual feature vector representing the global context of the image and a visual feature map preserving the spatial structure of the image in parallel based on the original visual features, include: dividing the original crop image into several local sub-images and scaling the original crop image into a panoramic reference image; extracting visual features of each local sub-image and the panoramic reference image based on the pre-trained image encoder; extracting and compressing the visual features of each local sub-image and the panoramic reference image through a pre-trained resampler based on a cross-attention mechanism to obtain extracted features and projecting them into the embedding space of a preset multimodal large language model to obtain a general visual feature vector representing the global context of the image; stitching the visual features of each local sub-image according to the spatial position of each local sub-image in the original crop image to obtain a high-resolution feature map; and performing bilinear interpolation upsampling on the visual features of the panoramic reference image to make its spatial size match the high-resolution feature map. Figure 1 The high-resolution feature map is then fused with the panoramic reference feature map element by element to obtain the visual feature map of the original crop image.
[0060] In one possible implementation, the steps of obtaining the region mask of the original crop image and sampling several sampling points from the foreground of the region mask, and obtaining the visual feature vectors corresponding to the sampling points in the visual feature map, include: obtaining the region mask of the original crop image and randomly sampling several sampling points from the foreground of the region mask, and normalizing the coordinates of each sampling point to obtain the normalized coordinates of each sampling point; wherein, the region mask of the original crop image is obtained by freely selecting on the visualized original crop image; and based on the normalized coordinates of each sampling point, the feature vectors corresponding to the normalized coordinates of each sampling point are extracted in the visual feature map by bilinear interpolation.
[0061] In one possible implementation, the step of performing local geometric topological aggregation of the visual feature vectors corresponding to several sampling points in the visual feature map to obtain the regional visual feature vector of the original crop image includes: sampling several backbone points from several sampling points using the farthest point sampling method; constructing a local geometric neighborhood for each backbone point based on several sampling points using the K-nearest neighbor algorithm, and obtaining the feature difference vector between the feature vectors of each neighbor point in the local geometric neighborhood of each backbone point and the feature vector of each backbone point, thereby obtaining several feature difference vectors for each backbone point; extracting the several feature difference vectors of each backbone point through a pre-trained fully connected layer to obtain the local gradient feature vector of each backbone point; concatenating the local gradient feature vectors and the visual feature vectors of each backbone point, and aggregating them through a max pooling layer to obtain the enhanced local feature vector of each backbone point; flattening the enhanced local feature vectors of all backbone points, and mapping them to a global feature vector through a pre-trained first linear projection layer; and projecting the global feature vector to a dimension consistent with the hidden layer of a preset multimodal large language model through a pre-trained second linear projection layer to obtain the regional visual feature vector of the original crop image.
[0062] In one possible implementation, the steps of acquiring the analysis question text, constructing a multimodal sequence by combining general visual feature vectors and regional visual feature vectors, and calling a preset multimodal large language model to obtain crop phenotypic analysis results include: encoding the analysis question text into text vectors, embedding the text vectors and regional visual feature vectors into preset multimodal sequence templates respectively, to obtain a text feature sequence fused with regional semantics; concatenating the general visual feature vectors with the text feature sequence fused with regional semantics to obtain a multimodal input sequence; and inputting the multimodal sequence into the preset multimodal large language model to obtain... The crop phenotypic analysis results; wherein, the preset multimodal large language model is obtained by training an improved multimodal large language model using a crop phenotypic analysis dataset; wherein, the improved multimodal large language model is obtained by replacing the last layer of the decoder of the multimodal large language model with a sparse hybrid expert layer containing several expert networks; wherein, the several expert networks of the sparse hybrid expert layer calculate the activation probability of each expert network according to the multimodal sequence through a preset routing network, and control the participation of a preset number of expert networks with high activation probabilities in the calculation through a Top-K gating mechanism based on the activation probability of each expert network.
[0063] In one possible implementation, when training the improved multimodal large language model using a crop phenotypic analysis dataset, full parameter fine-tuning is performed on the resampler, fully connected layer, first linear projection layer, and second linear projection layer based on the cross-attention mechanism; the decoder of the improved multimodal large language model is fine-tuned using a low-rank adaptive method; and the training loss function is an autoregressive language modeling loss function with added load balancing loss terms from several expert networks.
[0064] In one possible implementation, the crop phenotypic analysis dataset is constructed as follows: Several crop phenotypic analysis training image samples and key phenotypic expert annotation results for each sample are obtained; the key phenotypic expert annotation results for each training image sample are used as semantic anchors and embedded together with preset analysis question text samples into a preset large language model structured template to obtain strong constraint input prompts for each training image sample; the training image samples and their strong constraint input prompts are input into a multimodal large language model to obtain crop phenotypic analysis results for each training image sample; the training image samples, analysis question text samples, and crop phenotypic analysis results are integrated to form a crop phenotypic analysis dataset containing several sample pairs; wherein, a sample pair is a pair of crop phenotypic analysis training image samples, analysis question text samples, and crop phenotypic analysis results of the training image samples.
[0065] All relevant content of each step involved in the aforementioned embodiments of the crop phenotypic analysis method can be referenced to the functional description of the corresponding functional module of the crop phenotypic analysis system in the embodiments of the present invention, and will not be repeated here.
[0066] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.
[0067] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of crop phenotypic analysis methods.
[0068] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the crop phenotypic analysis method in the above embodiments.
[0069] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0070] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0071] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0072] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.< / unk> < / unk>
Claims
1. A method for analyzing crop phenotypic characteristics, characterized in that, include: Obtain the original crop image and extract the original visual features of the original crop image based on the pre-trained image encoder; Based on the original visual features, a general visual feature vector representing the global context of the image and a visual feature map that preserves the spatial structure of the image are generated in parallel. Obtain the region mask of the original crop image and sample several sampling points from the foreground of the region mask, and obtain the visual feature vectors corresponding to the sampling points in the visual feature map; Local geometric topological aggregation is performed on the visual feature vectors corresponding to several sampling points in the visual feature map to obtain the region visual feature vector of the original crop image. The system acquires the text of the analysis question, constructs a multimodal sequence by combining general visual feature vectors and regional visual feature vectors, and calls a preset multimodal large language model to obtain the phenotypic analysis results of crops.
2. The crop phenotypic analysis method according to claim 1, characterized in that, The process involves acquiring the original crop image and extracting the original visual features of the original crop image based on a pre-trained image encoder. Based on the original visual features, a general visual feature vector representing the global context of the image and a visual feature map preserving the spatial structure of the image are generated in parallel, including: The original crop image is divided into several local sub-images and scaled up to a panoramic reference image; visual features of each local sub-image and the panoramic reference image are extracted based on a pre-trained image encoder. By using a pre-trained resampler based on a cross-attention mechanism, the visual features of each local sub-image and the panoramic reference image are extracted and compressed to obtain the extracted features, which are then projected into the embedding space of a pre-defined multimodal large language model to obtain a general visual feature vector representing the global context of the image. The visual features of each local sub-image are stitched together according to their spatial position in the original crop image to obtain a high-resolution feature map; the visual features of the panoramic reference image are upsampled by bilinear interpolation to make their spatial size consistent with the high-resolution feature map to obtain a panoramic reference feature map; the high-resolution feature map and the panoramic reference feature map are fused element-wise to obtain the visual feature map of the original crop image.
3. The crop phenotypic analysis method according to claim 2, characterized in that, The steps of obtaining the region mask of the original crop image and sampling several sampling points from the foreground of the region mask, and obtaining the visual feature vectors corresponding to the several sampling points in the visual feature map, include: Obtain the region mask of the original crop image and randomly sample several sampling points from the foreground of the region mask. Normalize the coordinates of each sampling point to obtain the normalized coordinates of each sampling point. The region mask of the original crop image is obtained by freely selecting on the visualized original crop image. Based on the normalized coordinates of each sampling point, the feature vector corresponding to the normalized coordinates of each sampling point is extracted from the visual feature map by bilinear interpolation.
4. The crop phenotypic analysis method according to claim 3, characterized in that, The step of performing local geometric topological aggregation of the visual feature vectors corresponding to several sampling points in the visual feature map to obtain the region visual feature vector of the original crop image includes: Several backbone points are sampled from a number of sampling points using the farthest point sampling method; Based on several sampling points, the K-nearest neighbor algorithm is used to construct a local geometric neighborhood for each backbone point, and the feature vectors of each neighbor point in the local geometric neighborhood of each backbone point and the feature vectors of each backbone point are obtained to obtain several feature difference vectors of each backbone point; and the several feature difference vectors of each backbone point are extracted through a pre-trained fully connected layer to obtain the local gradient feature vectors of each backbone point. The local gradient feature vectors and visual feature vectors of each backbone point are concatenated and aggregated into enhanced local feature vectors of each backbone point through a max pooling layer; and the enhanced local feature vectors of all backbone points are flattened and mapped into global feature vectors through a pre-trained first linear projection layer. The global feature vector is projected through a pre-trained second linear projection layer to the same dimension as the hidden layer of the preset multimodal large language model to obtain the region visual feature vector of the original crop image.
5. The crop phenotypic analysis method according to claim 4, characterized in that, The process of acquiring and analyzing the problem text, constructing a multimodal sequence by combining general visual feature vectors and regional visual feature vectors, and calling a preset multimodal large language model to obtain crop phenotypic analysis results includes: The analysis question text is encoded into a text vector, and the text vector and the region visual feature vector are respectively embedded into a preset multimodal sequence template to obtain a text feature sequence that integrates region semantics; and the general visual feature vector is concatenated with the text feature sequence that integrates region semantics to obtain a multimodal input sequence. The multimodal sequences are input into a pre-defined multimodal large language model to obtain crop phenotypic analysis results; The preset multimodal large language model is obtained by training an improved multimodal large language model using a crop phenotypic analysis dataset; the improved multimodal large language model is obtained by replacing the last layer of the decoder of the multimodal large language model with a sparse hybrid expert layer containing several expert networks. Among them, several expert networks in the sparse hybrid expert layer calculate the activation probability of each expert network according to the multimodal sequence through a preset routing network, and control the participation of a preset number of expert networks with high activation probabilities in the calculation through a Top-K gating mechanism based on the activation probability of each expert network.
6. The crop phenotypic analysis method according to claim 5, characterized in that, When training the improved multimodal large language model using a crop phenotypic analysis dataset, full parameter fine-tuning is performed on the resampler, fully connected layer, first linear projection layer, and second linear projection layer based on the cross-attention mechanism; the decoder of the improved multimodal large language model uses a low-rank adaptive method for parameter fine-tuning; the training loss function is an autoregressive language modeling loss function with added load balancing loss terms from several expert networks.
7. The crop phenotypic analysis method according to claim 5, characterized in that, The crop phenotypic analysis dataset was constructed in the following manner: Acquire several crop phenotypic analysis training image samples and the key phenotypic expert annotation results of each crop phenotypic analysis training image sample; use the key phenotypic expert annotation results of each crop phenotypic analysis training image sample as semantic anchors, and embed them together with the preset analysis question text samples into the preset large language model structured template to obtain strong constraint input prompts for each crop phenotypic analysis training image sample. The training image samples for each crop phenotypic analysis, along with their strongly constrained input prompts, are input into a multimodal large language model to obtain the crop phenotypic analysis results for each training image sample. The training image samples, analysis question text samples, and the crop phenotypic analysis results are then integrated to form a crop phenotypic analysis dataset containing several sample pairs. Each sample pair consists of a pair of crop phenotypic analysis training image samples, an analysis question text sample, and the crop phenotypic analysis results of the crop phenotypic analysis training image samples.
8. A crop phenotypic analysis system, characterized in that, include: The extraction module is used to acquire the original crop image and extract the original visual features of the original crop image based on the pre-trained image encoder; Based on the original visual features, a general visual feature vector representing the global context of the image and a visual feature map that preserves the spatial structure of the image are generated in parallel. The sampling module is used to obtain the region mask of the original crop image and sample several sampling points from the foreground of the region mask, as well as obtain the visual feature vectors corresponding to the sampling points in the visual feature map. The aggregation module is used to perform local geometric topological aggregation on the visual feature vectors corresponding to several sampling points in the visual feature map to obtain the region visual feature vector of the original crop image. The analysis module is used to acquire the text of the analysis question, construct a multimodal sequence by combining general visual feature vectors and regional visual feature vectors, and call a preset multimodal large language model to obtain the phenotypic analysis results of crops.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the crop phenotyping method as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the crop phenotyping method as described in any one of claims 1 to 7.