Agricultural pest detection method based on multi-modal data fusion and large language model

Abstract

The application discloses a kind of agricultural pest detection method and device based on multi-modal data fusion and large language model, and is related to artificial intelligence technical field.The method comprises: obtaining the RGB image, hyperspectral image and text description information related to image of the crop to be detected;Constructing an agricultural pest detection model;Three different modal information is respectively input into the corresponding encoder, and the feature sequence of different modal is output;Cross-modal attention mechanism is used for cross-modal feature alignment, and the feature alignment process is optimized by calculating contrastive learning loss, and the three modal feature vectors after alignment are output, information fusion is carried out through multi-modal deep fusion module, and unified multi-modal feature representation is generated, and feature adapter is used to convert into embedding sequence;The embedding sequence is input into the large language model after fine-tuning for reasoning, and the pest category is output through the classification head.The accuracy and robustness of pest detection in complex agricultural scenarios can be improved by using the present application.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and apparatus for detecting agricultural pests and diseases based on multimodal data fusion and a large language model. Background Technology

[0002] Agricultural pests and diseases are key factors affecting crop yield and quality. Traditional pest and disease detection mainly relies on manual inspections and field diagnosis, which is not only time-consuming and labor-intensive but also highly subjective, easily missing the optimal time for prevention and control. With the development of computer vision technology, deep learning-based image recognition methods have been widely used in pest and disease detection. However, most of these methods rely on single RGB (visible light) images, which have limited accuracy and robustness when dealing with complex backgrounds, early lesions, or diseases with similar features.

[0003] To improve detection performance, researchers have begun to explore the introduction of multimodal data. For example, hyperspectral (HSI) images can provide rich spectral information that RGB images cannot capture, helping to identify physiological stresses in crops; text descriptions can provide prior knowledge related to diseases. However, existing multimodal fusion methods face several challenges: 1) In real agricultural scenarios, obtaining complete paired RGB, HSI, and text datasets is very difficult, and models need to be able to handle missing modal data; 2) Models may over-rely on information-rich text descriptions, forming a "modal shortcut" while neglecting deep learning of visual features; 3) There is still no mature solution for how to effectively fuse heterogeneous features from different modalities and utilize the powerful knowledge understanding and logical reasoning capabilities of the latest large language models (LLM) to assist in diagnosis. Summary of the Invention

[0004] To address the technical problems existing in the prior art, this invention provides a method and apparatus for detecting agricultural pests and diseases based on multimodal data fusion and a large language model. The technical solution is as follows: On the one hand, a method for detecting agricultural pests and diseases based on multimodal data fusion and a large language model is provided. This method is implemented by an agricultural pest and disease detection device based on multimodal data fusion and a large language model, and includes: S1. Acquire the RGB image, hyperspectral image, and text description information related to the image of the crop to be detected; S2. Construct an agricultural pest and disease detection model; wherein the model includes: an RGB encoder, an HSI encoder, a text encoder, a cross-modal alignment module, a multimodal deep fusion module, a feature adapter, a large language model with efficient parameter fine-tuning, and a classification head; S3. Input the RGB image into the RGB encoder to obtain the RGB image feature sequence; input the hyperspectral image into the HSI encoder to obtain the HSI image feature sequence; input the text description information into the text encoder to obtain the text feature sequence; S4. Input the RGB image feature sequence, HSI image feature sequence and text feature sequence into the cross-modal alignment module, use the cross-modal attention mechanism to align features of different modalities, optimize the feature alignment process by calculating the contrastive learning loss, and output the aligned feature vectors of the three modalities. S5. Input the aligned feature vectors of the three modalities into the multimodal deep fusion module for information fusion to generate a unified multimodal feature representation; S6. Input the multimodal feature representation into the feature adapter and convert the multimodal feature representation into an input embedding sequence that can be processed by a large language model; S7. Input the embedded sequence into a large language model with efficient parameter fine-tuning for inference, and output the final pest and disease category through the classification head.

[0005] On the other hand, an agricultural pest and disease detection device based on multimodal data fusion and a large language model is provided. This device is applied to an agricultural pest and disease detection method based on multimodal data fusion and a large language model. The device includes: The acquisition module is used to acquire the RGB image, hyperspectral image, and image-related text description information of the crop to be detected; A construction module is used to build an agricultural pest and disease detection model; wherein, the model includes: an RGB encoder, an HSI encoder, a text encoder, a cross-modal alignment module, a multimodal deep fusion module, a feature adapter, a large language model with efficient parameter fine-tuning, and a classification head; The feature extraction module is used to input the RGB image into an RGB encoder to obtain an RGB image feature sequence; input the hyperspectral image into an HSI encoder to obtain an HSI image feature sequence; and input the text description information into a text encoder to obtain a text feature sequence. The feature alignment module is used to input RGB image feature sequences, HSI image feature sequences, and text feature sequences into the cross-modal alignment module. It uses a cross-modal attention mechanism to align features of different modalities, optimizes the feature alignment process by calculating contrastive learning loss, and outputs the aligned feature vectors of the three modalities. The feature fusion module is used to input the aligned feature vectors of the three modalities into the multimodal deep fusion module for information fusion, generating a unified multimodal feature representation; The feature transformation module is used to input the multimodal feature representation into the feature adapter and convert the multimodal feature representation into an input embedding sequence that can be processed by a large language model. The inference module is used to input the embedded sequence into a large language model with efficient parameter fine-tuning for inference, and output the final pest and disease category through the classification head.

[0006] On the other hand, an agricultural pest and disease detection device based on multimodal data fusion and large language model is provided. The agricultural pest and disease detection device based on multimodal data fusion and large language model includes: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the methods in the above-mentioned agricultural pest and disease detection methods based on multimodal data fusion and large language model is implemented.

[0007] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement any of the above-described methods for agricultural pest and disease detection based on multimodal data fusion and large language models.

[0008] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: This invention combines complementary information from multiple data sources, leverages the powerful reasoning capabilities of large language models, and robustly handles the problem of missing modal data, significantly improving the accuracy and robustness of pest and disease detection in complex agricultural scenarios. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart of an agricultural pest and disease detection method based on multimodal data fusion and large language model provided by an embodiment of the present invention; Figure 2 This is a schematic diagram of the internal structure of a cross-modal alignment module provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the internal structure of a multimodal deep fusion module provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the internal structure of a feature adapter provided in an embodiment of the present invention; Figure 5This is a block diagram of an agricultural pest and disease detection device based on multimodal data fusion and a large language model, provided by an embodiment of the present invention. Figure 6 This is a schematic diagram of the structure of an agricultural pest and disease detection device based on multimodal data fusion and a large language model, provided in an embodiment of the present invention. Detailed Implementation

[0011] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0012] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0013] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0014] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0015] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0016] This invention provides a method for detecting agricultural pests and diseases based on multimodal data fusion and a large language model. This method can be implemented using an agricultural pest and disease detection device based on multimodal data fusion and a large language model, which can be a terminal or a server. Figure 1 The flowchart shown is for an agricultural pest and disease detection method based on multimodal data fusion and a large language model. The processing flow of this method may include the following steps: S1. Acquire the RGB image, hyperspectral image, and text description information related to the image of the crop to be detected.

[0017] In one feasible implementation, to enhance the model's adaptability in scenarios where hyperspectral modes are missing, embodiments of the present invention set hyperspectral mode enable control parameters.

[0018] The control parameters act on the forward propagation process of the agricultural pest and disease detection model, which includes an RGB encoder, an HSI encoder, a text encoder, a cross-modal alignment module, a multimodal fusion module, a feature adapter, a large language model, and a classification head.

[0019] Backpropagation is effective when the parameters of the RGB encoder, HSI encoder, and text encoder are not frozen. The large language model, after efficient parameter fine-tuning, freezes most of the main body during training, primarily updating LoRA parameters, and can retain a small number of specified layers for training according to configuration.

[0020] When the hyperspectral mode enable control parameter is true, the model inputs the hyperspectral image into the HSI encoder for feature extraction; When the hyperspectral mode enable control parameter is false, the model skips the HSI encoder and constructs a uniform all-zero feature sequence based on the dimension of the RGB feature sequence, which is used as the placeholder feature input for the hyperspectral mode to the subsequent cross-modal alignment module and multimodal fusion module.

[0021] When the hyperspectral branch is not enabled, the tensor dimension of the RGB feature sequence is first obtained, denoted as B×N×D, where B is the batch size, N is the feature sequence length, and D is the feature dimension. Next, a zero-dimensional feature sequence, also B×N×D, is constructed as a placeholder input for the hyperspectral modality, allowing subsequent cross-modal alignment and fusion processes to continue without changing the network structure. In the subsequent attention calculation and fusion modules, this zero tensor does not carry effective information from the hyperspectral modality, enabling the model to smoothly adapt to situations lacking hyperspectral data. Therefore, the entire model can smoothly degenerate from a trimodal system to a bimodal system without changing the network architecture or retraining.

[0022] In one feasible implementation, to alleviate the overfitting of the model to text data during training, a masking data augmentation strategy is implemented on the text description information during the training phase. Specifically: First, determine whether the current dataset is a training set and whether a text masking strategy is enabled; If both are true, a random number will be generated and compared with the preset occlusion probability. If the random number is less than the probability, the original text description containing rich pest and disease characteristics will be forcibly replaced with a generic template sentence that does not contain any discriminative information, such as "a picture of a plant leaf that needs to be analyzed." The purpose is to randomly "block" the text modality prompts during training, forcing the model to learn and mine key visual features from visual modalities such as RGB and HSI that are sufficient to distinguish different disease categories, thereby significantly enhancing the model's visual understanding ability and generalization performance.

[0023] S2. Construct an agricultural pest and disease detection model; the model includes: an RGB encoder, an HSI encoder, a text encoder, a cross-modal alignment module, a multimodal deep fusion module, a feature adapter, a large language model with efficient parameter fine-tuning, and a classification head.

[0024] Optionally, the multimodal deep fusion module includes a multimodal Transformer layer; wherein the fusion method includes hierarchical fusion or gated fusion; Among them, hierarchical fusion first performs preliminary fusion of two modal features through independent MLPs, and then splices intermediate features and outputs a unified multimodal feature representation through a deep MLP; Among them, gating fusion involves concatenating all modal features and inputting them into a gating network, adaptively generating weights for each modality, and then weighting them to obtain a unified multimodal feature representation.

[0025] S3. Input the RGB image into the RGB encoder to obtain the RGB image feature sequence; input the hyperspectral image into the HSI encoder to obtain the HSI image feature sequence; input the text description information into the text encoder to obtain the text feature sequence.

[0026] Optionally, the specific implementation process of S3 includes S31-S36: In this embodiment of the invention, the RGB encoder adopts a visual Transformer (ViT) based architecture.

[0027] S31. Input the RGB image into a block embedding layer, divide the image into multiple image blocks and perform linear embedding, and output the embedded image block sequence; S32. Concatenate the embedded image patch sequence with a learnable classification token and add position encoding to output the concatenated sequence; input the concatenated sequence into a multi-layer Transformer block for processing to output the classification token and patch token sequences; concatenate the classification token and patch token sequences to obtain the final RGB image feature sequence. S33. Input the hyperspectral image into a lightweight spectral-spatial joint processing network, perform depthwise classifiable convolution through an efficient spectral dimensionality reduction module to obtain a feature map after depthwise convolution; divide the channel dimensions of the feature map after depthwise convolution into multiple groups. Among them, the efficient spectral dimensionality reduction module can reduce the spectral dimension. This module combines depthwise separable convolution and spectral grouping attention. The depthwise separable convolution consists of a depthwise convolution and a pointwise convolution. The depthwise convolution performs spatial filtering independently on each input channel, and then the pointwise convolution (1x1 convolution) is responsible for information fusion in the channel dimension.

[0028] S34. Input the features of each group into the attention mechanism module and calculate the channel attention weights; multiply the channel attention weights by the features of the group and output the processed group features; concatenate all the processed group features to obtain the feature map enhanced by spectral attention. The attention mechanism module includes: global average pooling, two convolutional layers, the ReLU function, and the Sigmoid function.

[0029] S35. Reduce the spatial resolution of the feature map enhanced by spectral attention through a spatial downsampling block and input it into a multi-layer spectral-spatial joint processing block to calculate multi-head self-attention. Generate classification tokens through global average pooling and projection layers, and generate a tile token sequence through a block projection layer. Each layer of the spectral-spatial joint processing block contains a grouped spatial attention layer and a feedforward network.

[0030] In one feasible implementation, grouped spatial attention is calculated using multi-head self-attention in the spatial dimension, as expressed by the following formula (1): (1) Where Q represents the query, which is obtained by linear transformation of the input feature map; K represents the key, which is obtained by linear transformation of the input feature map; and V represents the value, which is obtained by linear transformation of the input feature map.

[0031] S36. Input the text description information into a pre-trained language model based on the Transformer architecture, and perform contextual semantic understanding on the input text through a multi-layer bidirectional Transformer encoder to output a text feature sequence.

[0032] Specifically, the hidden state of the last layer of the pre-trained language model is used as the text feature sequence, and the output corresponding to the [CLS] token is used as the pooling feature.

[0033] S4. Input the RGB image feature sequence, HSI image feature sequence and text feature sequence into the cross-modal alignment module, use the cross-modal attention mechanism to align features of different modalities, optimize the feature alignment process by calculating the contrastive learning loss, and output the three aligned modal feature vectors.

[0034] in, Figure 2 The diagram shown is a schematic diagram of the internal structure of a cross-modal alignment module provided in an embodiment of the present invention.

[0035] In one feasible implementation, the process of calculating attention through a cross-modal attention mechanism is represented by the following formula (2): (2) in, This represents the cross-modal enhanced feature sequence obtained by fusing contextual information from modality b with modality a as the query object. In other words, it is the feature representation formed by extracting relevant information from modality b with modality a as the query object. The characteristic sequence representing mode a; The characteristic sequence representing mode b; The query matrix represents the query obtained by learning linear mapping from the feature sequence of mode a, and is used to characterize the retrieval needs of other modal information at each feature position in mode a; The key matrix, obtained by learning linear mapping from the feature sequence of modality b, is used for similarity matching with the query matrix. This represents the value matrix obtained by learning a linear mapping from the feature sequence of mode b, used to provide content information of the weighted converged mode b; This represents the first learnable projection matrix; This represents the second learnable projection matrix; This represents the third learnable projection matrix; This represents the scale normalization factor determined by the feature dimensions of the query vector and key vector, used to scale the attention score and make the calculation of cross-modal attention weights more stable.

[0036] Optionally, the specific implementation process of S4 includes S41-S43: S41. Pool the RGB image feature sequence, HSI image feature sequence and text feature sequence respectively to obtain RGB global features, HSI global features and text global features; S42. Normalize the RGB global features, HSI global features, and text global features respectively to obtain normalized RGB features, normalized HSI features, and normalized text features. S43. Based on the normalized RGB features, normalized HSI features, and normalized text features, calculate the contrastive learning loss between the two modalities. Optimize the feature alignment process by calculating the contrastive learning loss, and output the aligned feature vectors of the three modalities. The aligned feature vectors of the three modalities include: the aligned RGB feature vector, the aligned HSI feature vector, and the aligned text feature vector.

[0037] Optionally, the process of calculating the contrastive learning loss between the two modalities is expressed by the following formula (3): (3) in, This represents the contrastive learning loss between two modalities; The batch size is denoted by ; i and j are the sample indices; a and b represent two different modalities; where a is RGB, then b is HSI or text; a is HSI, then b is RGB or text; a is text, then b is RGB or HSI. This indicates the calculation of cosine similarity. This refers to temperature hyperparameters. This represents the global feature vector obtained by the i-th sample after feature extraction, pooling, and normalization by the corresponding encoder under mode a; This represents the global feature vector obtained by the i-th sample after feature extraction, pooling, and normalization by the corresponding encoder under mode b.

[0038] The purpose of the contrastive learning loss function is to bring the different modal representations of the same disease sample closer together and push the representations of different samples further apart.

[0039] S5. Input the aligned feature vectors of the three modalities into the multimodal deep fusion module for information fusion to generate a unified multimodal feature representation.

[0040] Among them, such as Figure 3 The diagram shown is an internal structure schematic of a multimodal deep fusion module provided in an embodiment of the present invention. Aligned RGB feature vectors, aligned HSI feature vectors, and aligned text feature vectors are each given a modality-specific token. The RGB token, HSI token, and text token are concatenated into a sequence and input into a multimodal Transformer block for deep feature interaction. Updated RGB tokens, HSI tokens, and text tokens are extracted, and finally fused using hierarchical fusion or gated fusion strategies to output a unified multimodal feature representation.

[0041] In one feasible implementation, the aligned feature vectors of the three modalities, together with their respective learnable modality-specific tokens, are concatenated into a long sequence; the long sequence is then input into a network composed of stacked multimodal Transformer blocks for deep interaction; wherein each multimodal Transformer block is a standard Transformer encoder layer.

[0042] In one feasible implementation, multi-head self-attention and feedforward network calculations are performed on the input long sequence to obtain multimodal feature representations; wherein, the multi-head self-attention calculation process is represented by the following formula (4): (4) Where X represents the long sequence features of the input multimodal Transformer block, which is obtained by concatenating RGB modal features, HSI modal features, text modal features, and the learnable modal tokens corresponding to each modality; LayerNorm represents the normalization operation, which is used to normalize the input features in the feature dimension to stabilize the network training process; This represents the intermediate feature sequence obtained after multi-head self-attention computation and residual connection; This represents a multi-head self-attention computation function, used to model the relationships between different modal tokens and different feature positions within a long sequence.

[0043] In one feasible implementation, the computation process of the feedforward network is represented by the following formula (5): (5) in, This represents the multimodal fusion feature sequence output after processing by multi-head self-attention, residual connections, layer normalization, and feedforward network. This represents a feedforward neural network used to perform nonlinear transformations and feature enhancements on the feature representation of each token.

[0044] In one feasible implementation, after processing by the Transformer block, the corresponding token for each module is extracted; based on the corresponding token for each module, a hierarchical fusion or gated fusion strategy is used for final fusion; wherein, this embodiment of the invention uses a hierarchical fusion method for fusion, and the specific implementation process includes: The three modal tokens are input into the first layer and combined in pairs. They are then initially fused through independent MLPs to obtain the initial fusion features of RGB and HSI, the initial fusion features of RGB and text, and the initial fusion features of HSI and text. The initial fused features are input into the second layer for concatenation, and then finally fused through an MLP to output the fused features. In one feasible implementation, the input to the gating fusion strategy is tokens of three modalities; all modal tokens are concatenated and input into a gating network, i.e., MLP, which outputs the weights of each modality; based on the weights of each modality, the final output is a unified feature representation.

[0045] S6. Input the multimodal feature representation into the feature adapter and convert the multimodal feature representation into an input embedding sequence that can be processed by a large language model.

[0046] Among them, such as Figure 4 The figure shown is a schematic diagram of the internal structure of a feature adapter provided in an embodiment of the present invention.

[0047] Optionally, the specific implementation process of S6 includes S61-S63: S61. Initialize a set of learnable query vectors; use multimodal features as keys and values ​​in the cross-attention mechanism, and use the learnable query vectors as queries; S62. Calculate attention weights based on keys, values, and queries, and update the query vector to obtain an adapted feature sequence; In one feasible implementation, the feature adapter adopts a Q-Former-style architecture, which performs multiple cross-attention calculations with a set of learnable query vectors and multimodal features to extract and translate information. The cross-attention calculation process is represented by the following formula (6): (6) in, The adapted feature sequence output by the feature adapter is an intermediate feature representation formed by extracting key information from the multimodal fusion features using the learnable query vector, which is then projected into the embedding space of the large language model. It represents a set of learnable query vectors; Indicates key; Indicates the value.

[0048] S63. Input the obtained adapted feature sequence into a multilayer perceptron and project it into the embedding space of the large language model to obtain the final input embedding sequence.

[0049] In one feasible implementation, in order to achieve efficient fine-tuning of the parameters of a large language model, the embodiments of the present invention employ the low-rank adaptation (LoRA) method; wherein, the low-rank adaptation method connects a weight increment of low-rank decomposition representation in parallel with the original pre-trained weights of the large language model (which are kept frozen during training); In one feasible implementation, the large language model performs deep contextual understanding and reasoning on the input sequence, and finally takes the average pooling result of the last hidden state and inputs it into an MLP classification head composed of a linear layer, a normalization layer, an activation function and an output linear layer, and outputs the final probability of the pest and disease category.

[0050] S7. Input the embedded sequence into a large language model with efficient parameter fine-tuning for inference, and output the final pest and disease category through the classification head.

[0051] Optionally, the agricultural pest and disease detection model is optimized end-to-end using a joint loss function. The joint loss is composed of a weighted sum of classification cross-entropy loss and contrastive learning loss. The classification accuracy and modality alignment are simultaneously optimized by adjusting the weights of the two losses through balancing hyperparameters.

[0052] In one feasible implementation, the training process of this embodiment of the invention is optimized end-to-end by a joint loss function; wherein, the classification loss is used to measure the difference between the final prediction result of the model and the true label; the standard cross loss function is adopted, which is expressed by the following formula (7): (7) in, Classification loss; logits represents the output of the model's classification head; This represents the true category label of the sample. The classification loss directly drives the model's ability to distinguish between different categories of pests and diseases.

[0053] Among them, the contrastive learning loss guides the model to map features from different modalities to a unified semantic space, ensuring that the model can understand the intrinsic connections between different data sources.

[0054] In one feasible implementation, the joint loss function is expressed by the following formula (8): (8) in, Indicates joint loss; It is a hyperparameter used to balance the importance of classification and alignment tasks; This represents the contrastive learning loss. By minimizing the joint loss, the model learns classification while also ensuring effective alignment of multimodal features.

[0055] This invention combines complementary information from multiple data sources, leverages the powerful reasoning capabilities of large language models, and robustly handles the problem of missing modal data, significantly improving the accuracy and robustness of pest and disease detection in complex agricultural scenarios.

[0056] Figure 5This is a block diagram of an agricultural pest and disease detection device based on multimodal data fusion and a large language model, provided by an embodiment of the present invention. This device is used for an agricultural pest and disease detection method based on multimodal data fusion and a large language model. (Refer to...) Figure 5 The device includes an acquisition module 510, a construction module 520, a feature extraction module 530, a feature alignment module 540, a feature fusion module 550, a feature transformation module 560, and an inference module 570. Wherein: The acquisition module 510 is used to acquire the RGB image, hyperspectral image and text description information related to the image of the crop to be detected; Module 520 is used to build an agricultural pest and disease detection model; wherein, the model includes: an RGB encoder, an HSI encoder, a text encoder, a cross-modal alignment module, a multimodal deep fusion module, a feature adapter, a large language model with efficient parameter fine-tuning, and a classification head; The feature extraction module 530 is used to input the RGB image into an RGB encoder to obtain an RGB image feature sequence; input the hyperspectral image into an HSI encoder to obtain an HSI image feature sequence; and input the text description information into a text encoder to obtain a text feature sequence. The feature alignment module 540 is used to input the RGB image feature sequence, HSI image feature sequence and text feature sequence into the cross-modal alignment module, use the cross-modal attention mechanism to align features of different modalities, optimize the feature alignment process by calculating the contrastive learning loss, and output the aligned feature vectors of the three modalities. The feature fusion module 550 is used to input the aligned three modal feature vectors into the multimodal deep fusion module for information fusion to generate a unified multimodal feature representation; The feature conversion module 560 is used to input the multimodal feature representation into the feature adapter and convert the multimodal feature representation into an input embedding sequence that can be processed by a large language model. The reasoning module 570 is used to input the embedded sequence into a large language model with efficient parameter fine-tuning for reasoning, and output the final pest and disease category through the classification head.

[0057] Optionally, the feature extraction module 530 is used for: An RGB image is input into a block embedding layer, which divides the image into multiple image blocks and performs linear embedding, outputting a sequence of embedded image blocks. The embedded image patch sequence is concatenated with a learnable classification token and positional encoding is added to output the concatenated sequence; the concatenated sequence is input into a multi-layer Transformer block for processing, and the output is a sequence of classification tokens and patch tokens; the sequence of classification tokens and patch tokens is concatenated to obtain the final RGB image feature sequence. The hyperspectral image is input into a lightweight spectral-spatial joint processing network, and a depthwise classifiable convolution is performed through an efficient spectral dimensionality reduction module to obtain a feature map after depthwise convolution. The channel dimension of the feature map after depthwise convolution is divided into multiple groups. The features of each group are input into the attention mechanism module to calculate the channel attention weights; the channel attention weights are multiplied by the features of the group to output the processed group features; all processed group features are concatenated to obtain the feature map enhanced by spectral attention. The feature map enhanced by spectral attention is reduced in spatial resolution by a spatial downsampling block and then input into a multi-layer spectral-spatial joint processing block to calculate multi-head self-attention. Classification tokens are generated through global average pooling and projection layers, and a sequence of tile tokens is generated through a block projection layer. Textual description information is input into a pre-trained language model based on the Transformer architecture. The input text is then subjected to contextual semantic understanding through a multi-layer bidirectional Transformer encoder, and the output text feature sequence is generated.

[0058] Optionally, the feature alignment module 540 is used for: The RGB image feature sequence, HSI image feature sequence, and text feature sequence are pooled separately to obtain RGB global features, HSI global features, and text global features. The RGB global features, HSI global features, and text global features are normalized respectively to obtain normalized RGB features, normalized HSI features, and normalized text features. Based on the normalized RGB features, normalized HSI features, and normalized text features, the contrastive learning loss between the two modalities is calculated. The feature alignment process is optimized by calculating the contrastive learning loss, and the aligned feature vectors of the three modalities are output. The aligned feature vectors of the three modalities include: aligned RGB feature vector, aligned HSI feature vector, and aligned text feature vector.

[0059] Optionally, the process of calculating the contrastive learning loss between the two modalities is represented by the following formula (9): (9) in, This represents the contrastive learning loss between two modalities; The batch size is represented by i and j, which are the sample indices. a and b represent two different modalities. This indicates the calculation of cosine similarity. This refers to temperature hyperparameters. This represents the global feature vector obtained by the j-th sample after feature extraction, pooling, and normalization by the corresponding encoder under modality b; This represents the global feature vector obtained by the corresponding encoder after feature extraction, pooling, and normalization of the i-th sample in mode b.

[0060] Optionally, the feature conversion module 560 is used for; Initialize a set of learnable query vectors; use multimodal features as keys and values ​​in the cross-attention mechanism, and use the learnable query vectors as queries; Based on the key, value, and query, the attention weight is calculated and the query vector is updated to obtain the adapted feature sequence. The obtained adapted feature sequence is input into a multilayer perceptron and projected into the embedding space of a large language model to obtain the final input embedding sequence.

[0061] Optionally, the agricultural pest and disease detection model is optimized end-to-end using a joint loss function; wherein the joint loss is composed of a weighted sum of classification cross-entropy loss and contrastive learning loss, and the classification accuracy and modality alignment effect are simultaneously optimized by balancing the weights of the two losses through hyperparameter adjustment.

[0062] Optionally, the multimodal deep fusion module includes a multimodal Transformer layer; wherein the fusion method includes hierarchical fusion or gated fusion; Among them, hierarchical fusion first performs preliminary fusion of two modal features through independent MLPs, and then splices intermediate features and outputs a unified multimodal feature representation through a deep MLP; Among them, gating fusion involves concatenating all modal features and inputting them into a gating network, adaptively generating weights for each modality, and then weighting them to obtain a unified multimodal feature representation.

[0063] This invention combines complementary information from multiple data sources, leverages the powerful reasoning capabilities of large language models, and robustly handles the problem of missing modal data, significantly improving the accuracy and robustness of pest and disease detection in complex agricultural scenarios.

[0064] Figure 6 This is a schematic diagram of the structure of an agricultural pest and disease detection device based on multimodal data fusion and a large language model, provided in an embodiment of the present invention. Figure 6 As shown, agricultural pest and disease detection equipment based on multimodal data fusion and large language models can include the above-mentioned... Figure 5The illustrated agricultural pest and disease detection device is based on multimodal data fusion and a large language model. Optionally, the agricultural pest and disease detection device 610 based on multimodal data fusion and a large language model may include a first processor 2001.

[0065] Optionally, the agricultural pest and disease detection device 610 based on multimodal data fusion and large language model may also include a memory 2002 and a transceiver 2003.

[0066] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0067] The following is combined Figure 6 The components of the agricultural pest and disease detection device 610 based on multimodal data fusion and large language model are described in detail below: The first processor 2001 is the control center of the agricultural pest and disease detection device 610 based on multimodal data fusion and large language model. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0068] Optionally, the first processor 2001 can execute various functions of the agricultural pest and disease detection device 610 based on multimodal data fusion and large language model by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0069] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 6 CPU0 and CPU1 are shown in the diagram.

[0070] In a specific implementation, as one example, the agricultural pest and disease detection device 610 based on multimodal data fusion and large language models may also include multiple processors, for example... Figure 6The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0071] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0072] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or exist independently, and may be connected via the interface circuit of the agricultural pest and disease detection device 610 based on multimodal data fusion and large language model (…). Figure 6 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0073] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0074] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 6 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0075] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently, and can be connected to the interface circuit of the agricultural pest and disease detection device 610 based on multimodal data fusion and large language model. Figure 6(Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0076] It should be noted that, Figure 6 The structure of the agricultural pest and disease detection device 610 based on multimodal data fusion and large language model shown in the figure does not constitute a limitation on the router. The actual agricultural pest and disease detection device based on multimodal data fusion and large language model may include more or fewer components than shown, or combine some components, or have different component arrangements.

[0077] Furthermore, the technical effects of the agricultural pest and disease detection device 610 based on multimodal data fusion and large language model can be referred to the technical effects of the agricultural pest and disease detection method based on multimodal data fusion and large language model described in the above method embodiments, and will not be repeated here.

[0078] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or it may be any conventional processor, etc.

[0079] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0080] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0081] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0082] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0083] It should be understood that, in various embodiments of the present invention, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0084] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0085] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0086] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0087] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0088] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0089] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0090] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for detecting agricultural pests and diseases based on multimodal data fusion and a large language model, characterized in that, The method includes: S1. Acquire the RGB image, hyperspectral image, and text description information related to the image of the crop to be detected; S2. Construct an agricultural pest and disease detection model; wherein the model includes: an RGB encoder, an HSI encoder, a text encoder, a cross-modal alignment module, a multimodal deep fusion module, a feature adapter, a large language model with efficient parameter fine-tuning, and a classification head; S3. Input the RGB image into the RGB encoder to obtain the RGB image feature sequence; input the hyperspectral image into the HSI encoder to obtain the HSI image feature sequence; input the text description information into the text encoder to obtain the text feature sequence; S4. Input the RGB image feature sequence, HSI image feature sequence and text feature sequence into the cross-modal alignment module, use the cross-modal attention mechanism to align features of different modalities, optimize the feature alignment process by calculating the contrastive learning loss, and output the aligned feature vectors of the three modalities. S5. Input the aligned feature vectors of the three modalities into the multimodal deep fusion module for information fusion to generate a unified multimodal feature representation; S6. Input the multimodal feature representation into the feature adapter and convert the multimodal feature representation into an input embedding sequence that can be processed by a large language model; S7. Input the embedded sequence into a large language model with efficient parameter fine-tuning for inference, and output the final pest and disease category through the classification head.

2. The method for detecting agricultural pests and diseases based on multimodal data fusion and large language model according to claim 1, characterized in that, In step S3, the RGB image is input into the RGB encoder to obtain the RGB image feature sequence; The hyperspectral image is input into the HSI encoder to obtain the HSI image feature sequence; The text description information is input into the text encoder to obtain a text feature sequence, including: S31. Input the RGB image into a block embedding layer, divide the image into multiple image blocks and perform linear embedding, and output the embedded image block sequence; S32. Concatenate the embedded image patch sequence with a learnable classification token and add position encoding to output the concatenated sequence; input the concatenated sequence into a multi-layer Transformer block for processing to output the classification token and patch token sequences; concatenate the classification token and patch token sequences to obtain the final RGB image feature sequence. S33. Input the hyperspectral image into a lightweight spectral-spatial joint processing network, perform depthwise classifiable convolution through an efficient spectral dimensionality reduction module to obtain a feature map after depthwise convolution; divide the channel dimensions of the feature map after depthwise convolution into multiple groups. S34. Input the features of each group into the attention mechanism module and calculate the channel attention weights; multiply the channel attention weights by the features of the group and output the processed group features; concatenate all the processed group features to obtain the feature map enhanced by spectral attention. S35. Reduce the spatial resolution of the feature map enhanced by spectral attention through a spatial downsampling block and input it into a multi-layer spectral-spatial joint processing block to calculate multi-head self-attention. Generate classification tokens through global average pooling and projection layers, and generate a tile token sequence through a block projection layer. S36. Input the text description information into a pre-trained language model based on the Transformer architecture, and perform contextual semantic understanding on the input text through a multi-layer bidirectional Transformer encoder to output a text feature sequence.

3. The method for detecting agricultural pests and diseases based on multimodal data fusion and large language model according to claim 1, characterized in that, The S4 inputs the RGB image feature sequence, HSI image feature sequence, and text feature sequence into the cross-modal alignment module, uses a cross-modal attention mechanism to align features of different modalities, optimizes the feature alignment process by calculating contrastive learning loss, and outputs the aligned three modal feature vectors, including: S41. Pool the RGB image feature sequence, HSI image feature sequence and text feature sequence respectively to obtain RGB global features, HSI global features and text global features; S42. Normalize the RGB global features, HSI global features, and text global features respectively to obtain normalized RGB features, normalized HSI features, and normalized text features. S43. Based on the normalized RGB features, normalized HSI features, and normalized text features, calculate the contrastive learning loss between the two modalities. Optimize the feature alignment process by calculating the contrastive learning loss, and output the aligned feature vectors of the three modalities. The aligned feature vectors of the three modalities include: the aligned RGB feature vector, the aligned HSI feature vector, and the aligned text feature vector.

4. The method for detecting agricultural pests and diseases based on multimodal data fusion and large language model according to claim 3, characterized in that, The process of calculating the contrastive learning loss between the two modalities is expressed by the following formula (1): （1） in, This represents the contrastive learning loss between two modalities; The batch size is represented by i and j, which are the sample indices. a and b represent two different modalities. This indicates the calculation of cosine similarity. This refers to temperature hyperparameters. This represents the global feature vector obtained by the encoder, pooling, and normalization of the i-th sample under mode a; This represents the global feature vector obtained after encoder extraction, pooling, and normalization in mode b.

5. The method for detecting agricultural pests and diseases based on multimodal data fusion and large language model according to claim 1, characterized in that, In step S6, the multimodal feature representation is input into the feature adapter, and the multimodal feature representation is converted into an input embedding sequence that can be processed by a large language model. S51. Initialize a set of learnable query vectors; use multimodal features as keys and values ​​in the cross-attention mechanism, and use the learnable query vectors as queries; S52. Calculate attention weights based on keys, values, and queries, and update the query vector to obtain an adapted feature sequence. S53. Input the obtained adapted feature sequence into a multilayer perceptron and project it into the embedding space of the large language model to obtain the final input embedding sequence.

6. The method for detecting agricultural pests and diseases based on multimodal data fusion and large language model according to claim 1, characterized in that, The agricultural pest and disease detection model is optimized end-to-end using a joint loss function. The joint loss is composed of a weighted sum of classification cross-entropy loss and contrastive learning loss. The classification accuracy and modality alignment are simultaneously optimized by balancing the weights of the two losses through hyperparameter adjustment.

7. The method for detecting agricultural pests and diseases based on multimodal data fusion and large language model according to claim 1, characterized in that, The multimodal deep fusion module includes a multimodal Transformer layer; wherein the fusion method includes hierarchical fusion or gated fusion; Among them, hierarchical fusion first performs preliminary fusion of two modal features through independent MLPs, and then splices intermediate features and outputs a unified multimodal feature representation through a deep MLP; Among them, gating fusion involves concatenating all modal features and inputting them into a gating network, adaptively generating weights for each modality, and then weighting them to obtain a unified multimodal feature representation.

8. An agricultural pest and disease detection device based on multimodal data fusion and a large language model, wherein the agricultural pest and disease detection device based on multimodal data fusion and a large language model is used to implement the agricultural pest and disease detection method based on multimodal data fusion and a large language model as described in any one of claims 1-7, characterized in that, The device includes: The acquisition module is used to acquire the RGB image, hyperspectral image, and image-related text description information of the crop to be detected; A construction module is used to build an agricultural pest and disease detection model; wherein, the model includes: an RGB encoder, an HSI encoder, a text encoder, a cross-modal alignment module, a multimodal deep fusion module, a feature adapter, a large language model with efficient parameter fine-tuning, and a classification head; The feature extraction module is used to input the RGB image into an RGB encoder to obtain an RGB image feature sequence; input the hyperspectral image into an HSI encoder to obtain an HSI image feature sequence; and input the text description information into a text encoder to obtain a text feature sequence. The feature alignment module is used to input RGB image feature sequences, HSI image feature sequences, and text feature sequences into the cross-modal alignment module. It uses a cross-modal attention mechanism to align features of different modalities, optimizes the feature alignment process by calculating contrastive learning loss, and outputs the aligned feature vectors of the three modalities. The feature fusion module is used to input the aligned feature vectors of the three modalities into the multimodal deep fusion module for information fusion, generating a unified multimodal feature representation; The feature transformation module is used to input the multimodal feature representation into the feature adapter and convert the multimodal feature representation into an input embedding sequence that can be processed by a large language model. The inference module is used to input the embedded sequence into a large language model with efficient parameter fine-tuning for inference, and output the final pest and disease category through the classification head.

9. An agricultural pest and disease detection device based on multimodal data fusion and a large language model, characterized in that, The agricultural pest and disease detection equipment based on multimodal data fusion and large language model includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 7.

Images