A multi-modal diagnosis and classification method and system for crop diseases

Abstract

The application discloses a kind of crop disease multi-modal diagnosis and classification method and system, it is related to data processing technical field, the method of the present application includes collecting disease multi-modal data and pre-processing;Disease multi-modal diagnosis model is constructed, the image and text sequence in multi-modal data are respectively input into the visual feature extraction subnet and text feature coding subnet of disease multi-modal diagnosis model, extract global visual feature, sequence state feature and global text feature;Based on global visual feature and global text feature, calculate global matching score, perform fine-grained local interaction on global visual feature and sequence state feature, and introduce class channel attention mechanism to correct the features obtained by interaction, and generate multi-modal fusion features by weighting;Based on global visual feature, global text feature and multi-modal fusion feature, construct loss function and train disease multi-modal diagnosis model, input the data to be classified into disease multi-modal diagnosis model and output classification results.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for multimodal diagnosis and classification of crop diseases. Background Technology

[0002] Crop diseases are one of the main factors restricting agricultural production and affecting the quality and yield of agricultural products. Multimodal diagnosis and classification methods and systems for crop diseases refer to technologies that utilize deep learning techniques such as computer vision and natural language processing to comprehensively process image and text information about crop diseases. Through the fusion analysis of multiple modal data, intelligent identification and classification of disease types are achieved. This technology aims to simulate the cognitive approach of agricultural experts in the diagnostic process, that is, not only observing the external morphology of the disease but also combining relevant pathological characteristics for comprehensive judgment, thereby providing more reliable diagnostic results in complex field environments. Traditional disease diagnosis mainly relies on human experience, which suffers from strong subjectivity, low efficiency, and difficulty in large-scale promotion. Multimodal diagnostic technology based on deep learning can effectively overcome these drawbacks and provide scientific decision support for agricultural production.

[0003] Currently, several related technical solutions have been proposed in the field of crop disease diagnosis. For example, Chinese invention patent application number 202411286850.2 discloses a technical solution that, by introducing channel attention and position attention mechanisms and combining multi-scale feature fusion in the U-Net network, enhances the model's ability to extract image features to a certain extent. This solution can effectively handle the segmentation and severity quantification of crop leaf diseases, exhibits high accuracy in lesion region localization under a single visual dimension, and can provide specific disease severity classifications, thus possessing certain practical value. However… While the aforementioned existing technologies introduce attention mechanisms within the image, they lack the ability to fine-grainedly interact and align between visual and semantic features. Existing technologies only weight images in the spatial and channel dimensions, failing to establish a correspondence between local image regions and abstract semantic concepts. This lack of cross-modal fine-grained interaction in feature extraction makes it difficult for the model to understand the semantic connotations behind disease morphology. Existing technologies focus on pixel-level segmentation and area calculation. Although they can calculate the degree of disease, their feature extraction process mainly serves the segmentation task. For fine-grained classification tasks of disease types, their feature representation ability is insufficient. Summary of the Invention

[0004] The technical problem solved by this invention is that existing technologies lack fine-grained interaction and alignment capabilities between visual and semantic features. Existing technologies only weight images in the spatial and channel dimensions, failing to establish a correspondence between local image regions and abstract semantic concepts. This lack of cross-modal fine-grained interaction in feature extraction makes it difficult for models to understand the semantic connotations behind disease morphology. Existing technologies focus on pixel-level segmentation and area calculation. Although they can calculate the degree of disease, their feature extraction process mainly serves the segmentation task. For fine-grained classification tasks of disease types, their feature representation capabilities are insufficient.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a method for multimodal diagnosis and classification of crop diseases, comprising the following steps:

[0006] Step S1: Collect and preprocess multimodal data of the disease;

[0007] Step S2: Construct a multimodal disease diagnosis model. Input the images and text sequences in the multimodal data into the visual feature extraction subnet and text feature encoding subnet of the multimodal disease diagnosis model, respectively, to extract global visual features, sequence state features, and global text feature vectors.

[0008] Step S3: Calculate the global matching score based on global visual features and global text features, perform fine-grained local interaction on global visual features and sequence state features, and introduce a category channel attention mechanism to correct the features obtained from the interaction, and generate multimodal fusion features by weighting.

[0009] Step S4: Construct a loss function based on global visual features, global text features, and multimodal fusion features, and train the disease multimodal diagnosis model. Input the data to be classified into the disease multimodal diagnosis model and output the classification result.

[0010] Preferably, step S1 includes the following sub-steps:

[0011] The multimodal data includes disease image data, disease category labels, and corresponding disease text description data. The multimodal data is preprocessed to obtain standardized image and text sequences.

[0012] The disease image data includes images of crop leaves and stems;

[0013] The disease text description data includes morphological features of lesion color, shape and texture, and description of the distribution of lesions on the plant;

[0014] The preprocessing includes geometric transformation enhancement, color transformation enhancement, and size normalization of the disease image data, as well as text cleaning, word segmentation, and standardized encoding of the text description data.

[0015] Preferably, in step S2, the disease multimodal diagnostic model includes a visual feature extraction subnetwork and a text feature encoding subnetwork;

[0016] The specific process of constructing a visual feature extraction subnet and extracting global visual features includes:

[0017] The standardized image is passed through the four residual stages of the visual feature extraction subnet to generate a low-resolution feature map. The upsampled feature map is obtained by transposed convolution. The upsampled features of the upsampled feature map are fused with the shallow features of the corresponding resolution shallow feature map element by element through skip connections to generate an enhanced feature map.

[0018] The low-resolution feature map is the feature map output from the fourth residual stage;

[0019] The shallow features are the feature maps output by the second residual stage and the third residual stage;

[0020] An evolutionary nested receptive field structure is constructed, wherein the evolutionary nested receptive field structure is connected in a serial nesting manner, including a shallow projection stage, a nested expansion evolution stage, and a deep fusion stage;

[0021] The shallow projection stage includes a projection convolutional layer for dimensionality reduction of the enhanced feature map;

[0022] The nested dilatation evolution stage consists of three cascaded dilatational bottleneck structures with increasing dilatation rates. Each dilatational bottleneck structure contains depth-separable convolutional layers.

[0023] The deep fusion stage includes using 1×1 pointwise convolution to restore the number of feature channels after nested expansion to a preset dimension, and fusing all channel information;

[0024] The enhanced feature map is input into the evolutionary nested receptive field structure, and then sequentially undergoes shallow projection dimensionality reduction, nested dilated convolution to extract multi-scale context, and deep point convolution fusion to calculate global visual features.

[0025] Preferably, in step S2, the specific process of constructing the text feature encoding subnet and obtaining sequence state features and global text features includes:

[0026] The standardized text sequence is input into the pre-trained BERT model to obtain a word vector sequence;

[0027] A bidirectional long short-term memory network is constructed, comprising a forward layer and a backward layer, wherein each forward layer and the backward layer includes a gating unit and a hidden state transition matrix.

[0028] The word vector sequence is input into the bidirectional long short-term memory network, and the forward hidden state and backward hidden state at each time step are calculated using the gating unit and the hidden state transition matrix, respectively.

[0029] For each position in the word vector sequence, the calculated forward hidden state and backward hidden state are concatenated along the feature dimension to obtain the sequence state feature vector at that position. The sequence state feature vectors of all positions constitute the sequence state features.

[0030] The final state of the forward hidden state is concatenated with the final state of the backward hidden state to obtain the global text feature vector.

[0031] Preferably, in step S3, calculating the global matching score includes:

[0032] Global average pooling is performed on the global visual features to obtain a global visual feature vector. The cosine similarity between the global visual feature vector and the global text feature vector in the shared embedding space is calculated, and the global matching score is calculated.

[0033] Preferably, in step S3, the fine-grained local interaction includes text-guided image attention interaction and image-guided text attention interaction;

[0034] The text-guided image attention interaction uses the sequence state features as the query vector and the global visual feature vector as the key vector and value vector to calculate the attention weight of the text to the image region and generate text features that fuse visual information.

[0035] The image-guided text attention interaction uses the global visual feature vector as the query vector and the sequence state features as the key vector and value vector to calculate the attention weight of the image region to the text words and generate image features that fuse semantic information.

[0036] The text features and image features are fused to obtain the initial multimodal fusion features.

[0037] Preferably, in step S3, generating multimodal fusion features includes:

[0038] The category-channel attention mechanism includes parallel channel adaptive branches and category adaptive branches;

[0039] The channel adaptive branch includes a global average pooling layer, a dimension-reduced fully connected layer, and a dimension-upper fully connected layer connected in sequence.

[0040] The category adaptive branch includes a global average pooling layer, a category mapping convolutional layer, and an inverse projection layer connected in sequence;

[0041] The initial multimodal fusion feature input channel adaptive branch is subjected to global average pooling to obtain a multimodal fusion feature vector. The multimodal fusion feature vector is then passed sequentially through a dimension reduction fully connected layer, a nonlinear activation function, a dimension increase fully connected layer, and a Sigmoid activation function to generate a channel attention weight vector.

[0042] The initial multimodal fusion feature is input into the category adaptive branch and global average pooling is performed to obtain the initial multimodal fusion feature vector. The initial multimodal fusion feature vector is mapped to the category space through a category mapping convolutional layer to generate a category response vector. The category response vector is normalized by Softmax to obtain the category probability distribution. The category probability distribution is then mapped back to the channel dimension through an inverse projection layer to generate a category-induced channel weight vector.

[0043] The initial multimodal fusion feature vector is multiplied element-wise with the channel attention weight vector and the category-induced channel weight vector to output the multimodal fusion feature.

[0044] Preferably, in step S4, constructing the loss function includes:

[0045] Construct a multi-scale modal contrast loss function, which includes inter-modal contrast loss and intra-modal contrast loss;

[0046] The intermodal contrast loss includes a first intermodal contrast loss based on global visual feature vectors and global text feature vectors, and a second intermodal contrast loss based on multimodal fusion features;

[0047] The intramodal contrast loss includes a first intramodal contrast loss calculated based on global visual feature vectors and disease labels, and a second intramodal contrast loss calculated based on multimodal fusion features and disease labels;

[0048] The intermodal contrast loss is calculated as the ratio of the similarity index of positive sample pairs to the sum of the similarity indices of all sample pairs in a batch. The goal of the intermodal contrast loss is to maximize the similarity of matched image-text pairs in the feature space while minimizing the similarity of non-matching pairs in a batch.

[0049] The intramodal contrast loss is based on the disease category label and uses a supervised contrastive learning loss formula. For any anchor sample, other samples of the same category are defined as positive samples, and samples of different categories are defined as negative samples. The goal of the intramodal contrast loss is to maximize the proportion of the similarity between the anchor sample and all positive samples in the total similarity.

[0050] Preferably, in step S4, constructing the loss function further includes: the total loss function is a weighted sum of the multi-scale modality contrast loss function and the cross-entropy loss function of the disease classification task;

[0051] Based on the total loss function, the gradients of all learnable parameters in the multimodal diagnosis model of crop diseases are calculated using the backpropagation algorithm and iteratively updated. The training process is monitored using a validation set. Training stops when the classification accuracy of the validation set meets the preset convergence condition. The current state of the multimodal diagnosis model of diseases is saved as the trained multimodal diagnosis model of diseases. The disease data to be diagnosed is input into the trained model, and the crop disease classification results are output.

[0052] A multimodal diagnosis and classification system for crop diseases includes a data acquisition module, a feature extraction module, a fusion module, and a classification module;

[0053] The acquisition module is used to acquire and preprocess multimodal data of diseases;

[0054] The feature extraction module is used to construct a disease multimodal diagnosis model. It inputs the images and text sequences in the multimodal data into the visual feature extraction subnet and text feature encoding subnet of the disease multimodal diagnosis model, respectively, to extract global visual features, sequence state features and global text feature vectors.

[0055] The fusion module is used to calculate a global matching score based on global visual features and global text features, perform fine-grained local interactions on global visual features and sequence state features, and introduce a category channel attention mechanism to correct the features obtained from the interactions, and generate multimodal fusion features by weighting.

[0056] The classification module is used to construct a loss function based on global visual features, global text features, and multimodal fusion features, and to train the disease multimodal diagnosis model. The data to be classified is input into the disease multimodal diagnosis model and the classification result is output.

[0057] The beneficial effects of this invention are as follows: This invention provides a multimodal diagnosis and classification method and system for crop diseases. Compared with traditional single-modal or simple feature splicing techniques, this invention achieves deep fusion of visual features and textual semantics by constructing a deep interactive multimodal network. Its core innovation lies in proposing a category channel attention mechanism. This mechanism introduces category-induced prior constraints in the feature fusion stage. Unlike traditional channel attention, which only focuses on the statistical distribution of the features themselves, the category channel attention of this invention includes parallel channel adaptive branches and category adaptive branches. The category adaptive branch generates a category response vector and projects it back to the channel dimension to calculate the category-induced channel weights. This means that the model can dynamically tell the feature extraction network which specific feature channels to focus on based on the potential disease category prediction, thereby correcting and recalibrating the features obtained from fine-grained local interactions. This mechanism establishes a feedback loop from abstract categories to underlying feature channels, greatly enhancing the model's ability to capture key features of specific diseases, effectively suppressing the interference of background noise such as mud spot reflections, and significantly improving the diagnostic accuracy and robustness of crop diseases in complex field environments. Attached Figure Description

[0058] Figure 1 This is a flowchart illustrating the steps of a multimodal diagnosis and classification method for crop diseases provided in one embodiment of the present invention. Detailed Implementation

[0059] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0060] Example 1, referring to Figure 1 This paper provides a multimodal diagnosis and classification method for crop diseases, which includes the following steps:

[0061] Step S1: Collect and preprocess multimodal data of the disease;

[0062] Step S2: Construct a multimodal disease diagnosis model. Input the images and text sequences in the multimodal data into the visual feature extraction subnet and text feature encoding subnet of the multimodal disease diagnosis model, respectively, to extract global visual features, sequence state features, and global text feature vectors.

[0063] Step S3: Calculate the global matching score based on global visual features and global text features, perform fine-grained local interaction on global visual features and sequence state features, and introduce a category channel attention mechanism to correct the features obtained from the interaction, and generate multimodal fusion features by weighting.

[0064] Step S4: Construct a loss function based on global visual features, global text features, and multimodal fusion features, and train the disease multimodal diagnosis model. Input the data to be classified into the disease multimodal diagnosis model and output the classification result.

[0065] This invention proposes a complete end-to-end multimodal diagnostic process. By inputting image visual information and text description information into a dedicated subnet for extraction, and fusing them based on global matching scores and fine-grained local interactions, it can fully utilize the complementarity of multimodal data. This method overcomes the deficiency of single visual modality in identifying atypical disease features. By using text description as semantic navigation, it assists the model in locating lesions and improves the accuracy of disease diagnosis.

[0066] Step S1 includes the following sub-steps:

[0067] Multimodal data includes disease image data, disease category labels, and corresponding disease text description data. The multimodal data is preprocessed to obtain standardized image and text sequences.

[0068] Disease image data includes images of crop leaves and stems;

[0069] Disease text description data includes morphological characteristics of lesion color, shape and texture, and description of the distribution of lesions on the plant;

[0070] Preprocessing includes geometric transformation enhancement, color transformation enhancement, and size normalization of the disease image data, as well as text cleaning, word segmentation, and standardized encoding of the text description data.

[0071] In one specific embodiment of the present invention, crop disease data covering 10 crops and 27 diseases, totaling 61 categories, is obtained from the publicly available AI Challenger 2018 dataset. The data includes JPG format images and JSON files labeled with disease category tags. A standard morphological feature description is written for each disease category tag using artificial intelligence (e.g., the standard morphological feature description of corn rust is an elongated reddish-brown pustules clustered on both sides of the leaf). All disease category tags and corresponding morphological feature descriptions are organized into a JSON file as a disease knowledge base. When obtaining each piece of data from the AIChallenger 2018 dataset, the corresponding morphological feature description is queried in the disease knowledge base according to the disease category tag. The JPG format images, JSON files labeled with disease category tags, and corresponding morphological feature descriptions constitute a complete sample. All samples are divided into a training set and a validation set in an 8:2 ratio, and training batches are extracted from the training set.

[0072] The geometric transformation includes performing a 15-degree left rotation and a horizontal flip on the image. The color transformation includes increasing the saturation by 10% and decreasing the brightness by 5% to ensure that the color characteristics of the lesions remain recognizable under different lighting conditions. The image resolution is uniformly scaled and cropped to pixels, and the pixel values are normalized to the range [0, 1];

[0073] The preprocessing also includes cleaning the text description data, removing stop words and performing word segmentation, adding start tokens and separator tokens, and generating a standardized text sequence;

[0074] Removing stop words includes deleting words such as "de", "shi", "he", "zai" which have no actual semantic meaning, reducing noise interference;

[0075] Adding start tokens and separator tokens includes using a word segmentation tool to split the text into a Token sequence, adding a start token [CLS] at the head of the sequence to identify the start of the sequence and aggregate global features, and adding a separator token [SEP] at the end of the sequence to identify the end of the sentence, generating a standardized text sequence adapted to the pre-trained language model.

[0076] The present invention establishes a standardized data preprocessing mechanism. By performing geometric transformation and color transformation on the image, it simulates the changes in natural lighting and shooting angles in the field, enhancing the generalization ability of the model to the environment. At the same time, by establishing a disease knowledge base and writing standard morphological feature descriptions for each disease, cleaning the text and adding [CLS], [SEP] tokens, it ensures that the text sequence can be adapted to the pre-trained model and aggregate global semantics, laying a data foundation for subsequent high-quality feature alignment.

[0077] In step S2, the multi-modal disease diagnosis model includes a visual feature extraction subnet and a text feature encoding subnet;

[0078] The specific process of constructing the visual feature extraction subnet and extracting global visual features includes:

[0079] The standardized images are sequentially passed through four residual stages of the visual feature extraction subnet to generate low-resolution feature maps, obtain upsampled feature maps through transposed convolution, and perform element-wise fusion of the upsampled features of the upsampled feature maps and the shallow features of the shallow feature maps with corresponding resolutions through skip connections to generate enhanced feature maps;

[0080] The low-resolution feature map is the feature map output by the fourth residual stage;

[0081] The shallow features are the feature maps output by the second and third residual stages;

[0082] In a specific embodiment of the present invention, the preprocessed image data is input into the ResNet-50 backbone network. The preprocessed image data undergoes convolution and pooling, and the resolution is continuously reduced. The low-resolution image is gradually restored to its resolution through transposed convolution to obtain an upsampled feature map. The upsampled feature map contains high-level semantic information. The shallow features and the corresponding upsampled features are added element-wise through skip connections to fuse semantic and detail information.

[0083] In this embodiment, input The disease images are processed sequentially through four residual stages of the ResNet-50 backbone network: Stage 1, Stage 2, Stage 3, and Stage 4.

[0084] Disease images are processed by Conv1 ( Convolution kernel, stride=2, padding=3) and max pooling ( Pooling kernel (step=2, padding=1), output The image is then processed through Stage 1 (3 bottleneck blocks) for pure convolution to extract the output feature map. ,size With 256 channels, the feature map at this time... The output feature map is obtained after convolution with a stride of 2 through Stage 2 (4 bottleneck blocks). (Shallow features), size The channel count is 512. After passing through Stage 3 (6 bottleneck blocks) and convolution with a stride of 2, the output feature map is generated. ,size The channel count is 1024. After passing through Stage 4 (3 bottleneck blocks) and convolution with a stride of 2, the output feature map is generated. ,size Number of channels: 2048;

[0085] right Using transposed convolution, the size is magnified by 2 times, and then... The convolution channel number is adjusted to 1024, resulting in... upsampling features ;

[0086] Skip connections include the result obtained in the previous step and Adding elements one by one The fused feature map is obtained. ;

[0087] The transposed convolution is magnified by 2 times, and the number of channels is adjusted to 512. The resulting upsampled features are then added element-wise with the shallow features to obtain the final enhanced feature map.

[0088] An evolutionary nested receptive field structure is constructed, which is connected in a serial nesting manner, including a shallow projection stage, a nested expansion evolution stage, and a deep fusion stage.

[0089] The shallow projection stage includes projection convolutional layers used to reduce the dimensionality of the enhanced feature maps;

[0090] The nested dilatation evolution stage consists of three cascaded dilatational bottleneck structures with increasing dilatation rates. Each dilatational bottleneck structure contains depth-separable convolutional layers.

[0091] The deep integration phase includes utilizing Pointwise convolution restores the number of feature channels after nested expansion to the preset dimension and fuses all channel information;

[0092] The enhanced feature map is input into the evolutionary nested receptive field structure, and then successively passes through shallow projection dimensionality reduction, nested dilated convolution to extract multi-scale context, and deep point convolution fusion to calculate the global visual features.

[0093] In one specific embodiment of the present invention, the enhanced feature map is input into an evolutionary nested receptive field structure and processed in a serial nested manner, specifically including:

[0094] Input the enhanced feature map into a The convolutional layer (using the Hard-Swish activation function) reduces the number of channels from 512 to 256, outputting a projected feature map. Size retention ;

[0095] The nested expansion evolution stage consists of three cascaded expansion inverted bottleneck structures, namely Block A, Block B and Block C.

[0096] BlockA receives As input, using Depthwise separable convolution, with adjustable dilation rate. Padding is set to 2, output feature map Its receptive field covers the local lesion texture;

[0097] BlockB receives As input, using Depthwise separable convolution, with adjustable dilation rate. The output feature map is padded with 4. Its receptive field expands outward from the previous layer, covering the area surrounding the lesion;

[0098] BlockC receives As input, using Depthwise separable convolution, with adjustable dilation rate. Padding is set to 8, output feature map Its receptive field is further expanded, completely enclosing the receptive field range of the first two layers, capturing global semantics;

[0099] Will Enter one The pointwise convolutional layer restores the number of channels to 512, fuses the contextual information of all channels, and outputs the final global visual features.

[0100] An evolutionary nested receptive field structure was constructed, which uses a serial nesting method to connect expansion inverted bottleneck structures with different dilation rates. This design enables the network to obtain multi-scale receptive fields from small-scale textures to large-scale shapes while maintaining resolution. By fusing the contextual information of all channels through deep point convolution, the generated global visual feature map not only retains spatial location information but is also rich in deep semantics, effectively solving the problem that traditional convolutional networks are prone to losing small disease targets.

[0101] In step S2, the specific process of constructing the text feature encoding subnet and obtaining sequence state features and global text features includes:

[0102] The standardized text sequence is input into the pre-trained BERT model to obtain a word vector sequence;

[0103] A bidirectional long short-term memory network is constructed, which includes a forward layer and a backward layer. Both the forward and backward layers contain gating units and hidden state transition matrices.

[0104] The word vector sequence is input into a bidirectional long short-term memory network, and the forward hidden state and backward hidden state at each time step are calculated using gating units and hidden state transition matrices, respectively.

[0105] For each position in the word vector sequence, the calculated forward hidden state and backward hidden state are concatenated along the feature dimension to obtain the sequence state feature vector at that position. The sequence state feature vectors of all positions constitute the sequence state features.

[0106] The final state of the forward hidden state is concatenated with the final state of the backward hidden state to obtain the global text feature vector.

[0107] In one specific embodiment of the present invention, a text sequence (e.g., sequence S=[CLS], leaf, now, bluish-gray, spot, [SEP], the sequence length is 6) is input into a pre-trained BERT model. The pre-trained BERT model uses the bert-base-chinese model (Chinese basic version) released by Google. The BERT model contains a 12-layer Transformer encoder neural network to perform deep computation and contextual interaction on the input text sequence. After computation, the output size is [size missing]. The matrix, i.e., the word vector sequence ;

[0108] The vector formed by the elements of the first row of the matrix corresponds to the start marker [CLS] of the text sequence, and the vector formed by the elements of the last row of the matrix corresponds to the separator marker [SEP] at the end of the text sequence. Except for the first and last rows, the vector formed by each row of the matrix corresponds to the deep semantics of specific words (such as cyan) in the text sequence.

[0109] The mathematical expression for a word vector sequence is:

[0110] ;

[0111] in Indicates the first in the sequence Word vectors at each position;

[0112] Using the feedforward layer of a bidirectional LSTM network, following the forward order of the sequence (i.e., from sequence...) arrive Process the word vector sequences sequentially;

[0113] The word vector sequence is input into a 512-dimensional LSTM forward layer. The LSTM forward layer segments the word vector sequence row by row. At the logical time step... At that time, extract the first row of the word vector sequence ( ), at logical moment At that time, extract the second row of the word vector sequence ( ), and so on, at logical moments At that time, extract the sixth line of the word vector sequence ( );

[0114] In processing the first Word vectors at each position At that time, the LSTM forward layer combines the current input... Forward hidden state of the previous position Perform calculations and output the forward hidden state at the current position. ;

[0115] In this embodiment, when At that time, read in ([CLS]), generate the forward hidden state. ;

[0116] when At that time, read in (Blue-gray), combined with previous memory, generate a forward hidden state. (At this point, the vector contains the semantic meaning of the leaf appearing bluish-gray).

[0117] when At that time, read in ([SEP]), generate the forward hidden state. ;

[0118] Using the backward layer of a bidirectional LSTM network, following the reverse order of the sequence (i.e., from the sequence...) arrive Process word vectors sequentially;

[0119] The LSTM backward layer splits the word vector sequence in reverse order row by row, at the logical time step. When extracting the sixth line of the word vector sequence (corresponding to the last line of the physical input),... ), at logical moment At that time, extract the fifth line of the word vector sequence ( ), and so on, at logical moments At that time, extract the first row of the word vector sequence ( );

[0120] In processing the first Word vectors at each position At that time, the backward layer combines the current input The backward hidden state of the next position Perform calculations and output the backward hidden state at the current position. ;

[0121] In this embodiment, when At that time, read in ([SEP]) generates the backward hidden state. ;

[0122] when At that time, read in (Blue-gray), combined with the memory of the following text, generates a backward hidden state. (At this point, the vector includes the following modifying semantics of the bluish-gray spots).

[0123] when At that time, read in ([CLS]), generate the backward hidden state. ;

[0124] For each position in the sequence (from arrive The calculated forward hidden state With backward hidden state Concatenate along the feature dimension;

[0125] The splicing formula is: ;

[0126] in, This represents the forward hidden state at position t. It is in a backward hidden state;

[0127] Each state has 512 dimensions, after splicing It has 1024 dimensions;

[0128] All of The sequence state feature matrix is ​​obtained by concatenation. Its size is This is used for subsequent fine-grained interactions with the image region;

[0129] Extract the final state after the LSTM feedforward layer has processed the entire sequence. ) and the final state after the backward LSTM processes the entire sequence ( By concatenating these elements, we obtain a dimension that is... The global text feature vector, which aggregates bidirectional contextual information of the entire text, is used for coarse-grained global matching.

[0130] This invention uses the BERT model combined with a bidirectional LSTM network to extract text features. The BERT model provides deep word embedding vectors, while the bidirectional LSTM, through gating units and hidden state transition matrices, can simultaneously capture the contextual dependencies of text sequences. The resulting sequence state features can accurately correspond to each word in the text, providing rich and ordered semantic information for subsequent fine-grained alignment of text image regions.

[0131] In step S3, calculating the global matching score includes:

[0132] Global average pooling is performed on the global visual features to obtain the global visual feature vector. The cosine similarity between the global visual feature vector and the global text feature vector in the shared embedding space is calculated, and the global matching score is calculated.

[0133] In one specific embodiment of the present invention, global average pooling is performed on the global visual feature map (512 dimensions) to obtain the global visual feature vector. ;

[0134] The dimension of the shared embedded space is set as follows ,Will Input consists of fully connected layers The visual projection layer, consisting of an activation function, outputs a mapped visual vector. ( ),in Shape ;

[0135] global text feature vector (1024-dimensional) input is supplied by a fully connected layer. The text projection layer, consisting of the activation function, yields the mapped text vector. ,in, Shape ;

[0136] calculate and The cosine similarity in the shared space serves as the global matching score, which represents the degree of matching between the current input image and the input text in coarse-grained semantics. This score will be used as the weighting basis for subsequent feature fusion or classification decisions.

[0137] This invention obtains a global matching score by calculating the cosine similarity between global visual features and global text features in a shared embedding space. This step provides a coarse-grained semantic filtering mechanism before feature fusion to measure the overall consistency between the image and the text description. This score serves as the weight basis for subsequent fusion or classification, helping the model to quickly eliminate obviously mismatched image-text pairs and improve inference efficiency and accuracy.

[0138] In step S3, fine-grained local interaction includes text-guided image attention interaction and image-guided text attention interaction;

[0139] Text-guided image attention interaction uses sequence state features as query vectors and global visual feature vectors as key and value vectors to calculate the attention weight of text to image regions and generate text features that fuse visual information.

[0140] Image-guided text attention interaction uses global visual feature vectors as query vectors and sequence state features as key and value vectors to calculate the attention weights of image regions to text words, generating image features that fuse semantic information.

[0141] Text features and image features are fused to obtain initial multimodal fusion features.

[0142] In one specific embodiment of the present invention, a system comprising Layer (in this embodiment) The bidirectional stacked deep attention interaction structure is used to perform fine-grained interactions. Each layer of the bidirectional stacked deep attention interaction structure includes two independent computing units: a first multi-head cross attention unit and a second multi-head cross attention unit. The bidirectional stacked deep attention interaction structure has a total of 2N units (4 units in this embodiment).

[0143] The first multi-head cross-attention unit is configured as a text-guided image attention interaction unit. This unit initializes three sets of independent learnable weight parameters, which are specifically used to calculate the degree of attention of text features to image regions and inject image information into text features.

[0144] The second multi-head cross-attention unit is configured as an image-guided text attention interaction unit. This unit also initializes three sets of independent learnable weight parameters, which are specifically used to calculate the degree of attention of the image region to the text words and inject the text semantics into the image features.

[0145] The global visual feature vector output in step S2 is called. In this embodiment, the global visual feature vector (size) is used. Flattened in the spatial dimension, reorganized into... A set of local region feature vectors , where the set of vectors It is a 512-dimensional vector, representing the nth element in the image. Texture features of individual grid regions (such as the region corresponding to the location of lesions in the middle of the leaf);

[0146] Each feature vector maps back to a tiny 8*8 pixel patch in the original image in physical space. For early crop diseases, lesions are often tiny, pinpoint-shaped or scattered. Traditional global features would drown out this tiny lesion information with a large area of ​​healthy leaves. However, this invention retains the 28*28 dimension, so that each tiny lesion, even if it only occupies 1% of the image area, has an independent feature vector.

[0147] Call the sequence state features output in step S3 and define them as being composed of... A set of word feature vectors , where the set of vectors A 1024-dimensional vector (assuming Bi-LSTM output dimension), representing the th element in the text sequence. The contextual semantic features of a word (e.g., the word corresponding to "blue-gray");

[0148] In this embodiment, the bidirectional stacked deep attention interaction structure includes a first interaction layer and a second interaction layer, each layer including a first multi-head cross attention unit and a second multi-head cross attention unit;

[0149] The set of feature vectors of a local region The set consisting of word feature vectors Performing the first interaction layer operations specifically includes:

[0150] Using the first multi-head cross-attention unit, image information is aggregated into text features through mathematical operations. The first multi-head cross-attention unit contains three pre-set learnable weight matrices, which are the query weight matrix and... Key weight matrix Sum weight matrix ;

[0151] The hidden layer dimension of the interactive space is set to ,but The size is (Used to convert text vectors) and All dimensions are (Used to transform image vectors), the values ​​in these matrices are random numbers when initialized, and are continuously updated and optimized during training through the backpropagation algorithm;

[0152] Text feature set and Perform matrix multiplication to generate the query matrix. (size ), to set image features respectively with and Perform matrix multiplication to generate the key matrix. (size Sum-value matrix (size );

[0153] Perform matrix multiplication The resulting size is The original score matrix is ​​used to perform a Softmax normalization operation on each row of the matrix to obtain the attention weight distribution matrix;

[0154] When the text sequence processes the keyword "pustule", the model will automatically calculate the similarity between the word vector and 784 regions in the image. For small lesion areas scattered throughout the leaves, the matching score will be significantly higher than that of the surrounding healthy leaf areas.

[0155] It is beneficial to achieve semantic navigation effect, that is, to use detailed descriptions in the text to indicate small lesion areas that are difficult to notice in complex leaf backgrounds, solve the problem that small target diseases are easily lost in traditional convolutional neural network technology, and realize fine-grained alignment between the text description and disease pixels constructed by this invention.

[0156] The attention weight distribution matrix and the value matrix Perform matrix multiplication and output a first-order text feature matrix that incorporates visual information. (size is) At this point, each word vector in the text sequence has incorporated the visual information of its corresponding region in the image;

[0157] By utilizing a second multi-head cross-attention unit, explicit semantic labels are assigned to image regions through mathematical operations. The second multi-head cross-attention unit internally contains three additional learnable weight matrices, namely the query weight matrix. Key weight matrix Sum weight matrix ;

[0158] The size is (Used to convert image vectors) and All dimensions are (Used to convert text vectors);

[0159] Image feature set and Multiply to generate the query matrix (size ), to set text features respectively with and Multiplication generates a key matrix. (size Sum-value matrix (size );

[0160] Perform matrix multiplication And perform Softmax normalization to obtain a size of Attention weight distribution matrix;

[0161] The weight matrix and the value matrix Perform matrix multiplication and output a first-order image feature matrix that incorporates semantic information. (size is) The matrix retains the spatial structure features of the image in physical form, but the feature vector of each pixel has been fused with the corresponding text semantics.

[0162] In this invention, Each local feature vector in the image is not only a visual texture, but also an enhanced feature highlighted by the text semantics. Specifically, for small lesions that are originally blurry in the image, if there is a corresponding feature description in the text, the feature value of that region will be amplified during the fusion process; conversely, background noise such as mud spots and reflections that are unrelated to the text description are suppressed.

[0163] right and Perform the second interaction layer operation using the same method to obtain the second-order text feature matrix. and second-order image feature matrix ;

[0164] That is to For querying, with Using the key and value, the text vector is scanned again for the first-order enhanced image features, outputting a second-order text feature matrix. ;

[0165] by For querying, with Using the key and value, the image region vector is scanned again for the first-order enhanced text features, outputting a second-order image feature matrix. (size );

[0166] Since the ultimate goal of this invention is the identification and localization of diseases, spatial dimension information must be preserved. Therefore, a second-order image feature matrix is ​​selected. As the processing object (discarding the text matrix which has no spatial information), the second-order image feature matrix Its input source By performing element-wise addition, the residual fusion characteristics are obtained: This step is to prevent gradient vanishing and preserve the original visual information;

[0167] Will The input is a feedforward neural network, which includes: linear layer 1, ReLU activation, and linear layer 2, used to perform non-linear transformations on the features, integrate feature fragments generated by the multi-head attention mechanism, enhance the expressive power of the features, and output the matrix (size) of the feedforward neural network. Reshape the image according to the spatial arrangement of the original image to restore its original size. The three-dimensional tensor is the initial multimodal fusion feature.

[0168] This invention designs a bidirectional stacked deep attention interaction structure that performs text-guided image attention and image-guided text attention interaction. This bidirectional interaction mechanism realizes semantic navigation. With text as the query, it can highlight tiny areas in the image that match descriptions such as "pustule" and "spot". With image as the query, it can give text words specific visual perception. By fusing second-order features through residual connections and feedforward networks, it not only preserves the original visual information and prevents gradient vanishing, but also incorporates strong semantic information into each pixel, significantly enhancing the discriminative power of the features.

[0169] In step S3, generating multimodal fusion features includes:

[0170] The category-channel attention mechanism includes parallel channel-adaptive branches and category-adaptive branches;

[0171] The channel adaptive branch consists of a globally average pooling layer, a dimension-reduced fully connected layer, and a dimension-upper fully connected layer connected in sequence.

[0172] The class-adaptive branch consists of a global average pooling layer, a class-mapping convolutional layer, and an inverse projection layer connected in sequence;

[0173] The initial multimodal fusion feature input channel adaptive branch is subjected to global average pooling to obtain the multimodal fusion feature vector. The multimodal fusion feature vector is then passed sequentially through a dimension reduction fully connected layer, a non-linear activation function, a dimension increase fully connected layer, and a Sigmoid activation function to generate the channel attention weight vector.

[0174] The initial multimodal fusion feature is input into the category adaptive branch and global average pooling is performed to obtain the initial multimodal fusion feature vector. The initial multimodal fusion feature vector is mapped to the category space through the category mapping convolutional layer to generate the category response vector. The category response vector is normalized by Softmax to obtain the category probability distribution. The category probability distribution is then mapped back to the channel dimension through the inverse projection layer to generate the category-induced channel weight vector.

[0175] The initial multimodal fusion feature vector is multiplied element-wise with the channel attention weight vector and the category-induced channel weight vector to output the multimodal fusion feature.

[0176] In a specific embodiment of the present invention, the initial multimodal fusion feature is denoted as... Its size is (In this embodiment, it is) ), for initial multimodal fusion features Perform global average pooling to reduce the spatial dimension Compress to 1 to obtain the initial multimodal fusion feature vector. ;

[0177] Initial multimodal fusion feature vector The mathematical expression is:

[0178] ;

[0179] in, This indicates the initial multimodal fusion features in coordinates. Pixel feature value at that location, The height of the initial multimodal fusion feature. The width of the multimodal fusion feature. This represents the total number of pixels on the initial multimodal feature. It is to calculate the average. This indicates a horizontal traversal, adding elements from column 1 to column 2 of each row. List, This indicates a vertical traversal, adding each column from row 1 to row 2. OK;

[0180] The initial multimodal fusion features are processed through a dimensionality-reduction fully connected layer, a non-linear activation function, a dimensionality-upgrading fully connected layer, and a sigmoid activation function to generate channel attention weight vectors, specifically including:

[0181] The initial multimodal fusion feature vector Input to a dimension-reducing fully connected layer, multiplied by a weight matrix (size In this embodiment, the scaling ratio is taken. That is, the weight matrix Size is The feature vectors are then processed by the ReLU activation function to obtain dimensionality-reduced feature vectors. These dimensionality-reduced feature vectors are then input into the up-dimensional fully connected layer and multiplied by the weight matrix. (size Obtain the upgraded feature vector, apply the Sigmoid activation function to the upgraded feature vector, and map the values ​​to... Interval, output channel attention weight vector (size );

[0182] The initial multimodal fusion feature vector Enter one Convolutional layer (as classifier), with the number of convolutional kernels set to... , thus obtaining a dimension of The class response vector (i.e., the category response vector), where the value represents which class the current image belongs to. The original score of the disease;

[0183] The mathematical expression for calculating the category response vector is:

[0184] ;

[0185] in, This is the initial multimodal fusion feature vector. Here is the weight matrix of the convolutional layer. For category response vectors;

[0186] The number of convolution kernels is equal to the total number of disease classification categories. ;

[0187] Perform a softmax operation on the class response vector to obtain the class probability distribution vector. (size );

[0188] Will Input inverse projection layer (the inverse projection layer is a fully connected layer, weight matrix) Size is ), obtain the category-induced channel weight vector (size );

[0189] Category-induced channel weight vector The mathematical expression is:

[0190] ;

[0191] in, The channel weight vector is induced by the category. This is the weight matrix of the inverse projection layer;

[0192] The channel attention weight vectors calculated from the two branches above, the category-induced channel weights, and the original features are fused to obtain multimodal fusion features;

[0193] The formula for calculating fusion is:

[0194] ;

[0195] in This represents element-wise multiplication under the broadcast mechanism.

[0196] This invention creatively introduces a category-channel attention mechanism that includes two branches: channel adaptation and category adaptation. In particular, the category adaptation branch generates category response vectors by mapping features to the category space and then projecting them back to the channel dimension to generate category-induced channel weights. This mechanism enables the model to enhance the feature channels (such as specific color or texture channels) that are unique to the initially predicted disease category, thereby greatly improving the accuracy of fine-grained classification.

[0197] In step S4, constructing the loss function includes:

[0198] Construct a multi-scale modal contrast loss function, which includes inter-modal contrast loss and intra-modal contrast loss;

[0199] The inter-modal contrast loss includes a first inter-modal contrast loss based on global visual feature vectors and global text feature vectors, and a second inter-modal contrast loss based on multimodal fusion features;

[0200] The intramodal contrast loss includes a first intramodal contrast loss calculated based on global visual feature vectors and disease labels, and a second intramodal contrast loss calculated based on multimodal fusion features and disease labels;

[0201] Intermodal contrast loss is calculated as the ratio of the similarity index of positive sample pairs to the sum of the similarity indices of all sample pairs within a batch. The goal of intermodal contrast loss is to maximize the similarity of matched image-text pairs in the feature space while minimizing the similarity of non-matching pairs within a batch.

[0202] The intramodal contrast loss is based on the disease category label and uses a supervised contrastive learning loss formula. For any anchor sample, other samples of the same category are defined as positive samples, and samples of different categories are defined as negative samples. The goal of the intramodal contrast loss is to maximize the proportion of the similarity between the anchor sample and all positive samples in the total similarity.

[0203] In a specific embodiment of the present invention, the mathematical expression for the inter-modal contrast loss is:

[0204] ;

[0205] in, For the contrast loss between first modes, This represents the total number of samples in the training batch. , They represent the first Global visual feature vector and global text feature vector (positive sample pair) of each sample. , Indicates the first in the batch Global text feature vector and global visual feature vector of each sample (when (when it is a negative sample) This represents the function for calculating cosine similarity. This represents the temperature coefficient, used to adjust the smoothness of the probability distribution (set to 0.07 in this embodiment). Indicates the first The cosine similarity between the global visual feature vector and the global text feature vector of a sample is the global matching score. Indicates the first The global visual feature vector of the nth sample and the nth sample Cosine similarity between global text feature vectors of each sample Indicates the first The global text feature vector of the nth sample and the nth sample Cosine similarity between global visual feature vectors of each sample;

[0206] Indicates the first The global visual feature vector of each sample and the global text feature vectors of all samples in the training batch (including positive samples) and all negative samples The sum of similarity indices of ( );

[0207] Indicates the first The sum of the similarity indices between the global text feature vector of each sample and all global visual feature vectors (including positive samples and all negative samples) in the training batch;

[0208] The training batch is a pre-built training set containing N samples. Each sample contains a disease image, a disease text description, and a disease category label.

[0209] Intermodal contrast loss The mathematical expression is:

[0210] ;

[0211] in, For the contrast loss between the second modes, This represents the total number of samples in the training batch. , They represent the first Image features that fuse semantic information and text features that fuse visual information for each sample. , Indicates the first in the training batch Image features that fuse semantic information and text features that fuse visual information for each sample (when...) (when it is a negative sample) This represents the function for calculating cosine similarity. This represents the temperature coefficient, used to adjust the smoothness of the probability distribution (set to 0.07 in this embodiment).

[0212] The first-modal intra-modal contrast loss includes both image-side and text-side aspects. The mathematical expression for the image-side first-modal intra-modal contrast loss is:

[0213] ;

[0214] in, This represents the contrast loss within the first modality of the image. Indicates the first The global visual feature vector of each sample. This represents the set of positive sample indices, that is, all samples in the current batch that are related to the first sample. A set of indices for other samples that have the same disease category label. Represents a set The number of elements in the middle is used for normalized averaging. Represents a set The global visual feature vector of the sample in the sample;

[0215] This represents the set of indices for all comparison samples, i.e., the set of indices excluding the first sample in the current batch. The index of all samples other than the current sample. Represents a set The global visual feature vector of any sample in the dataset. Represents the natural exponential function;

[0216] The mathematical expression for the text-side intramodal contrast loss is:

[0217] ;

[0218] in, This represents the contrast loss within the first modality on the text side. Indicates the first The global text feature vector of each sample. This represents the set of positive sample indices, that is, all samples in the current batch that are related to the first sample. A set of indices for other samples that have the same disease category label. Represents a set The number of elements in the middle is used for normalized averaging. Represents a set The global text feature vector of the sample in the sample;

[0219] This represents the set of indices for all comparison samples, i.e., the set of indices excluding the first sample in the current batch. The index of all samples other than the current sample. Represents a set The global text feature vector of any sample in the dataset. Represents the natural exponential function;

[0220] Intramodal contrast loss Image-side intramodal contrast loss Contrast loss within the first modality of the text side The sum of;

[0221] The mathematical expression for the intramodal contrast loss is:

[0222] ;

[0223] in, This represents the contrast loss within the second mode. Indicates the first Multimodal fusion features of each sample This represents the set of positive sample indices, that is, all samples in the current batch that are related to the first sample. A set of indices for other samples that have the same disease category label. Represents a set The number of elements in the middle is used for normalized averaging. Represents a set Multimodal fusion features of samples in the dataset;

[0224] This represents the set of indices for all comparison samples, i.e., the set of indices excluding the first sample in the current batch. The index of all samples other than the current sample. Represents a set Multimodal fusion features of any sample in the dataset. This represents the natural exponential function.

[0225] This invention constructs a multi-scale modal contrast loss function that includes inter-modal and intra-modal contrast losses. By bringing positive sample pairs closer together and pushing negative sample pairs further apart, the model learns more discriminative representations in the feature space. This contrastive learning strategy not only optimizes the alignment of images and text but also enhances intra-class compactness and inter-class separability, further improving classification performance.

[0226] In step S4, constructing the loss function also includes: the total loss function is a weighted sum of the multi-scale modality contrast loss function and the cross-entropy loss function of the disease classification task;

[0227] Based on the total loss function, the gradients of all learnable parameters in the multimodal diagnosis model of crop diseases are calculated and iteratively updated using the backpropagation algorithm. The training process is monitored using a validation set. Training stops when the classification accuracy of the validation set meets the preset convergence condition. The current state of the multimodal diagnosis model of diseases is saved as the trained multimodal diagnosis model of diseases. The disease data to be diagnosed is input into the trained model, and the crop disease classification results are output.

[0228] In a specific embodiment of the present invention, the mathematical expression for the final total loss function is:

[0229] ;

[0230] in, This represents the total loss value ultimately used for backpropagation to update parameters in the network. The cross-entropy loss function represents the task of disease classification. For the contrast loss between first modes, For the contrast loss between the second modes, For the intramodal contrast loss, For the intramodal contrast loss, and To balance the weighting coefficients, the proportion of loss at different scales in the total loss is adjusted (in this embodiment, we take...). );

[0231] Let represent the cross-entropy loss function for the disease classification task. The mathematical expression for the cross-entropy loss function for the disease classification task is:

[0232] ;

[0233] in, This represents the total number of samples in the training batch. This indicates the total number of categories of crop diseases. This represents the index number of the sample in the training batch. Index number indicating the type of disease. Indicates the first The sample at the th The true label value for each category (the true label value is a one-hot encoded value, which is 1 if the sample belongs to the category, and 0 otherwise). The model predicts the first... The sample belongs to the first The probability value of each disease category (i.e., the output of the multimodal fusion features after processing by a fully connected layer and a Softmax function);

[0234] The set of network parameters to be optimized includes: all convolutional kernel weights of the ResNet-50 backbone network, projective convolutional weights in the shallow projection stage and depth-separable convolutional weights in the nested dilation evolution stage of the evolutionary nested receptive field structure, gate weights (forget gate, input gate, and output gate weights) and hidden state transition matrices within the bidirectional LSTM network, and query weight matrix, key weight matrix, and value weight matrix (i.e., ...) in fine-grained local interactions. The weights of the fully connected layer in the channel adaptive branch and the classifier weight matrix in the category adaptive branch;

[0235] Based on the calculated total loss value Using the backpropagation algorithm, the partial derivatives (i.e., gradients) of the total loss with respect to each network parameter are calculated layer by layer according to the chain rule. );

[0236] Based on the calculated gradient direction, adjust the parameter values ​​in the opposite direction of the gradient:

[0237] ;

[0238] in, The learning rate (in this embodiment, it is initially set to...) ).

[0239] An early stopping strategy is adopted to determine the time point for the optimization to end. After each training cycle, the current model is tested using the validation set, and the disease classification accuracy on the validation set is calculated. If the validation set accuracy does not improve or decreases within 10 consecutive cycles, it is determined that the model has reached the optimal convergence state. At this time, training is terminated immediately, and the network parameters with the minimum loss value are used as the final model parameters.

[0240] The process of calculating the disease classification accuracy on the validation set includes:

[0241] Input the samples in the validation set, and output the probability distribution vector of each image belonging to each disease category. Select the category with the highest probability as the predicted disease category label of the image. Compare the predicted label with the real category labels recorded in the validation set one by one, count the number of correctly predicted samples, and take the ratio of the number of correctly predicted samples to the total number of samples in the validation set as the disease classification accuracy.

[0242] The disease data to be diagnosed includes images of crop leaves or stems, and text descriptions selected by the user regarding the color, shape, and distribution of lesions. This data must undergo the exact same preprocessing as during the training phase before being input into the model.

[0243] The crop disease classification results refer to the disease category labels and confidence levels output by the model after calculation.

[0244] This invention constructs a composite total loss function, which weights and fuses the cross-entropy loss and multi-scale modal contrast loss of the disease classification task. This joint optimization design realizes dual supervision of feature learning and classification decision. The cross-entropy loss directly constrains the output of the model to ensure the accuracy of the disease classification results. Meanwhile, the introduction of multi-scale modal contrast loss as a regularization term forces the model to bring matching image-text pairs closer and push away mismatched pairs in the deep feature space, thereby learning robust features with higher semantic alignment and more significant inter-class differences.

[0245] A multimodal diagnosis and classification system for crop diseases includes a data acquisition module, a feature extraction module, a fusion module, and a classification module;

[0246] The data acquisition module is used to collect and preprocess multimodal data on diseases;

[0247] The feature extraction module is used to construct a multimodal disease diagnosis model. It inputs the images and text sequences in the multimodal data into the visual feature extraction subnet and text feature encoding subnet of the multimodal disease diagnosis model, respectively, to extract global visual features, sequence state features and global text feature vectors.

[0248] The fusion module is used to calculate the global matching score based on global visual features and global text features, perform fine-grained local interactions on global visual features and sequence state features, and introduce a category channel attention mechanism to correct the features obtained from the interactions, and generate multimodal fusion features by weighting.

[0249] The classification module is used to construct a loss function based on global visual features, global text features, and multimodal fusion features, and to train the disease multimodal diagnosis model. The data to be classified is input into the disease multimodal diagnosis model and the classification result is output.

[0250] This invention provides a multimodal diagnosis and classification method and system for crop diseases. Compared with traditional single-modal or simple feature splicing techniques, this invention achieves deep fusion of visual features and textual semantics by constructing a deep interactive multimodal network. Its core innovation lies in proposing a category-channel attention mechanism. This mechanism introduces category-induced prior constraints in the feature fusion stage. Unlike traditional channel attention, which only focuses on the statistical distribution of features themselves, the category-channel attention of this invention includes parallel channel adaptive branches and category adaptive branches. The category adaptive branch generates a category response vector and projects it back to the channel dimension to calculate the category-induced channel weights. This means that the model can dynamically tell the feature extraction network which specific feature channels to focus on based on the potential disease category prediction, thereby correcting and recalibrating the features obtained from fine-grained local interactions. This mechanism establishes a feedback loop from abstract categories to underlying feature channels, greatly enhancing the model's ability to capture key features of specific diseases, effectively suppressing the interference of background noise such as mud spot reflections, and significantly improving the diagnostic accuracy and robustness of crop diseases in complex field environments.

[0251] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0252] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multi-modal diagnosis and classification method for crop diseases, characterized in that, The method comprises the following steps: Step S1, collecting disease multi-modal data and preprocessing; Step S2, constructing a disease multi-modal diagnosis model, inputting the image and text sequence in the multi-modal data into the visual feature extraction subnetwork and the text feature encoding subnetwork of the disease multi-modal diagnosis model respectively, and extracting global visual feature, sequence state feature and global text feature vector; Step S3, calculating the global matching score based on the global visual feature and the global text feature, performing fine-grained local interaction on the global visual feature and the sequence state feature, and introducing a class channel attention mechanism to correct the features obtained by interaction to generate multi-modal fusion features; Step S4, constructing a total loss function based on the global visual feature, the global text feature and the multi-modal fusion feature, and training the disease multi-modal diagnosis model, and inputting the data to be classified into the disease multi-modal diagnosis model to output the classification result; The fine-grained local interaction includes text-guided image attention interaction and image-guided text attention interaction; The text-guided image attention interaction is to take the sequence state feature as a query vector, take the global visual feature vector as a key vector and a value vector, calculate the attention weight of the text to the image region, and generate text features fused with visual information; The image-guided text attention interaction is to take the global visual feature vector as a query vector, take the sequence state feature as a key vector and a value vector, calculate the attention weight of the image region to the text vocabulary, and generate image features fused with semantic information; The text features and image features are fused to obtain initial multi-modal fusion features; The class channel attention mechanism includes a parallel channel adaptive branch and a class adaptive branch; The channel adaptive branch includes a global average pooling layer, a dimension reduction fully connected layer and a dimension increasing fully connected layer connected in sequence; The class adaptive branch includes a global average pooling layer, a class mapping convolution layer and an inverse projection layer connected in sequence.

2. The crop disease multi-modal diagnosis and classification method as claimed in claim 1, wherein, The step S1 comprises the following substeps: The multi-modal data includes disease image data, disease category label and corresponding disease text description data, and the multi-modal data is preprocessed to obtain standardized images and text sequences; The disease image data includes images of crop leaves and stem parts; The disease text description data includes morphological feature descriptions of disease spot color, shape and texture, and descriptions of the distribution of disease spots on the plant; The preprocessing includes geometric transformation enhancement, color transformation enhancement and size normalization processing on the disease image data, and text cleaning, word segmentation and standardization coding processing on the text description data.

3. The crop disease multi-modal diagnosis and classification method as claimed in claim 2, wherein, In step S2, the disease multi-modal diagnosis model includes a visual feature extraction subnetwork and a text feature encoding subnetwork; The specific process of constructing the visual feature extraction subnetwork and extracting the global visual feature includes: The standardized image is sequentially subjected to four residual stages of a visual feature extraction subnetwork to generate a low-resolution feature map, an up-sampling feature map is obtained through transposed convolution, and the up-sampling features of the up-sampling feature map are element-wise fused with shallow features of a shallow feature map of a corresponding resolution through a skip connection to generate an enhanced feature map; The low-resolution feature map is a feature map output by a fourth residual stage; The shallow features are feature maps output by a second residual stage and a third residual stage; An evolutionary nested receptive field structure is constructed, the evolutionary nested receptive field structure is connected in a serial nested manner, and includes a shallow projection stage, a nested expansion evolution stage, and a deep fusion stage; The shallow projection stage includes a projection convolution layer for dimension reduction of the enhanced feature map; The nested expansion evolution stage is composed of three expansion inverted bottle neck structures connected in series, the expansion rates of which show an increasing trend, and each expansion inverted bottle neck structure internally includes a depth separable convolution layer; The deep fusion stage comprises using The point-by-point convolution restores the number of feature channels after nested expansion to a preset dimension and fuses all channel information. The enhanced feature map is input into the evolutionary nested receptive field structure, sequentially subjected to shallow projection dimension reduction, nested expansion convolution for multi-scale context extraction, and deep point convolution fusion, and global visual features are calculated.

4. The crop disease multi-modal diagnosis and classification method of claim 3, wherein, In the step S2, the specific process of constructing a text feature encoding subnetwork and obtaining sequence state features and global text features includes: The standardized text sequence is input into a pre-trained BERT model to obtain a word vector sequence; A bidirectional long short-term memory network is constructed, the bidirectional long short-term memory network includes a forward layer and a backward layer, and the forward layer and the backward layer both internally include a gating unit and a hidden state conversion matrix; The word vector sequence is input into the bidirectional long short-term memory network, and the forward hidden state and the backward hidden state of each time step are calculated using the gating unit and the hidden state conversion matrix, respectively; For each position in the word vector sequence, the calculated forward hidden state and backward hidden state are spliced in the feature dimension to obtain a sequence state feature vector of the position, and sequence state features of all positions constitute sequence state features; The final state of the forward hidden state and the final state of the backward hidden state are spliced to obtain a global text feature vector.

5. The crop disease multi-modal diagnosis and classification method as claimed in claim 4, wherein, In the step S3, calculating the global matching score includes: The global visual features are subjected to global average pooling to obtain a global visual feature vector, the cosine similarity of the global visual feature vector and the global text feature vector in a shared embedding space is calculated, and the global matching score is calculated.

6. The crop disease multi-modal diagnosis and classification method as claimed in claim 5, wherein, In the step S3, generating the multi-modal fusion feature includes: The initial multi-modal fusion feature is input into a channel adaptive branch to obtain a multi-modal fusion feature vector through global average pooling, and the multi-modal fusion feature vector is sequentially subjected to a dimension reduction fully connected layer, a nonlinear activation function, a dimension increasing fully connected layer, and a Sigmoid activation function to generate a channel attention weight vector; The initial multi-modal fusion feature is input into a category adaptive branch for global average pooling to obtain an initial multi-modal fusion feature vector, the initial multi-modal fusion feature vector is mapped to a category space through a category mapping convolution layer to generate a category response vector, the category response vector is subjected to Softmax normalization processing to obtain a category probability distribution, and the category probability distribution is mapped back to a channel dimension through an inverse projection layer to generate a category-induced channel weight vector; The initial multi-modal fusion feature vector is multiplied element by element with the channel attention weight vector and the category-induced channel weight vector to output a multi-modal fusion feature.

7. The crop disease multi-modal diagnosis and classification method as claimed in claim 6, wherein, In the step S4, the loss function is constructed, including: a multi-scale modal contrast loss function is constructed, the loss function including an inter-modal contrast loss and an intra-modal contrast loss; the inter-modal contrast loss includes a first inter-modal contrast loss based on the global visual feature vector and the global text feature vector and a second inter-modal contrast loss based on the multi-modal fusion feature; the intra-modal contrast loss includes a first intra-modal contrast loss based on the global visual feature vector and the disease label and a second intra-modal contrast loss based on the multi-modal fusion feature and the disease label; the inter-modal contrast loss is a ratio of a similarity index of a positive sample pair to a sum of similarity indexes of all sample pairs in a batch, and the target of the inter-modal contrast loss is to maximize the similarity of a matched image-text pair in a feature space while minimizing the similarity of non-matched pairs in the batch; the intra-modal contrast loss is based on the disease category label, and a supervised contrast learning loss formula is used, for any anchor sample, other samples of the same category are defined as positive samples, and samples of different categories are defined as negative samples, and the target of the intra-modal contrast loss is to maximize the proportion of the similarity between the anchor sample and all positive samples in the total similarity.

8. The crop disease multi-modal diagnosis and classification method as claimed in claim 7, wherein, In the step S4, the total loss function is also constructed, including that the total loss function is a weighted sum of the multi-scale modal contrast loss function and a cross-entropy loss function of the disease classification task; based on the total loss function, the gradients of all learnable parameters in the crop disease multi-modal diagnosis model are calculated using a back propagation algorithm and are iteratively updated, the training process is monitored using a validation set, the training is stopped when the classification accuracy of the validation set meets a preset convergence condition, the current state of the disease multi-modal diagnosis model is saved as a trained disease multi-modal diagnosis model, and the disease data to be diagnosed is input into the trained model to output a crop disease classification result.

9. A crop disease multi-modal diagnosis and classification system, characterized in that, The system comprises a collection module, a feature extraction module, a fusion module, and a classification module. The collection module is configured to collect and preprocess disease multi-modal data. The feature extraction module is configured to construct a disease multi-modal diagnosis model, input an image and a text sequence in multi-modal data into a visual feature extraction subnetwork and a text feature encoding subnetwork of the disease multi-modal diagnosis model respectively, and extract a global visual feature, a sequence state feature, and a global text feature vector. The fusion module is configured to calculate a global matching score based on the global visual feature and the global text feature, perform fine-grained local interaction on the global visual feature and the sequence state feature, introduce a class channel attention mechanism to correct the feature obtained through the interaction, and generate a multi-modal fusion feature through weighting; The classification module is configured to construct a loss function based on the global visual feature, the global text feature, and the multi-modal fusion feature, train the disease multi-modal diagnosis model, input the data to be classified into the disease multi-modal diagnosis model, and output a classification result; the fine-grained local interaction includes text-guided image attention interaction and image-guided text attention interaction; The text-guided image attention interaction is to take the sequence state feature as a query vector, take the global visual feature vector as a key vector and a value vector, calculate attention weights of the text on the image region, and generate a text feature fused with visual information; The image-guided text attention interaction is to take the global visual feature vector as a query vector, take the sequence state feature as a key vector and a value vector, calculate attention weights of the image region on the text vocabulary, and generate an image feature fused with semantic information; The text feature and the image feature are fused to obtain an initial multi-modal fusion feature; The class channel attention mechanism includes a parallel channel adaptive branch and a class adaptive branch; The channel adaptive branch includes a global average pooling layer, a dimension reduction full connection layer, and a dimension increase full connection layer connected in sequence; The class adaptive branch includes a global average pooling layer, a class mapping convolution layer, and an inverse projection layer connected in sequence.

Images