Lightweight rice disease identification method based on multi-modal low-rank bilinear pooling

Abstract

The invention discloses a lightweight rice disease identification method based on multi-modal low-rank bilinear pooling, which comprises the following steps: firstly, carrying out data acquisition and data set construction, then constructing a lightweight multi-modal identification model, respectively obtaining image features and spectral features through a double-branch structure, and finally, carrying out multi-modal low-rank bilinear multi-modal fusion through a low-rank bilinear multi-modal fusion module. Image and spectral features are fully fused to obtain joint representation features, and rice disease recognition is realized. According to the invention, efficient and accurate identification of rice canopy diseases and pests is realized by fusing visible light images and spectral features. According to the method, a lightweight neural network structure is adopted, the model complexity and computing resource consumption are effectively reduced while the recognition precision is ensured, and the method is suitable for being deployed in a smart agriculture scene with limited hardware resources.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of intelligent agriculture, and specifically relates to a lightweight rice disease identification method based on multi-modal low-rank bilinear pooling. The algorithm combines deep learning and multi-modal feature processing technology, and adopts a low-rank bilinear pooling mechanism to fuse key features in visible light images and spectral data, thereby achieving efficient and accurate identification of rice diseases and being suitable for resource-limited intelligent agricultural scenarios. BACKGROUND

[0002] Rice diseases and pests are key risk factors threatening global food security. According to FAO data, about 100 million tons of rice are lost annually due to diseases and pests worldwide. As a major rice-producing country, China's rice production can be reduced by 20%-50% due to major diseases such as rice blast and chilo suppressalis. Existing rice disease detection technologies are mostly limited to the application of single modalities of images or spectra. Image recognition-based methods have the advantages of low equipment cost, simple operation, and rapid data acquisition, and can intuitively capture disease symptoms through visible light images, making them suitable for large-area rapid screening. However, this method is significantly affected by environmental factors (such as changes in light and occlusion interference), and has insufficient generalization ability in complex field backgrounds, resulting in high false detection rates. Multi-spectral technology-based methods have the potential for early disease identification and can detect disease before symptoms are visible, providing key information for timely prevention and control. However, the high-dimensional data processing of this technology requires strong computing power support, resulting in significantly increased equipment cost and operational complexity, making it difficult to meet the real-time monitoring needs of agricultural production lines.

[0003] In view of the limitations of single modal technology, combining the advantages of image recognition and multi-spectral technology to develop image-spectrum fusion technology is a potential direction for improving the efficiency of rice disease and pest detection. The technology disclosed in application number CN202311165338.8 is a multi-modal fusion detection method for rice leaf diseases suitable for edge computing scenarios, which combines image and spectral features to provide higher accuracy in rice disease detection. However, this method is complex and cannot train and deploy the model end-to-end. SUMMARY

[0004] To address the deficiencies in the prior art, the present application provides a lightweight rice canopy disease identification method based on deep learning and multi-modal low-rank bilinear pooling, which realizes efficient and accurate identification of rice canopy diseases and pests by fusing visible light images and spectral features. The method uses a lightweight neural network structure to effectively reduce model complexity and computing resource consumption while ensuring identification accuracy, making it suitable for deployment in hardware resource-limited intelligent agricultural scenarios.

[0005] The method can improve the detection accuracy and early identification capability while maintaining low cost and easy operation, and provides new technical support for accurate prevention and control of rice diseases and pests.

[0006] The light-weight rice disease identification method based on multi-modal low-rank bilinear pooling comprises the following steps:

[0007] Step 1: Data acquisition and construction of data set.

[0008] The RGB image and the non-imaging spectral data of 530nm, 570nm, 680nm, 700nm, 740nm and 780nm of the rice canopy are collected to obtain a 6-dimensional non-imaging spectral vector, and registration and preprocessing are performed to construct a multi-modal data set for training and evaluation.

[0009] Step 2: Construct a light multi-modal identification model.

[0010] The model includes a double-branch structure composed of an image feature extraction module and a spectral feature extraction module, and a low-rank bilinear multi-modal fusion module.

[0011] Step 3: Obtain image features and spectral features through the double-branch structure.

[0012] Step 4: Fully fuse image and spectral features through the low-rank bilinear multi-modal fusion module to obtain joint representation features and realize rice disease identification.

[0013] In one possible implementation, the image feature extraction module adopts a four-stage hierarchical structure. Each stage includes a down-sampling layer and an SE-Star Block module, which effectively extracts the features of the RGB image through this hierarchical design . Among them, the SE-Star Block combines the advantages of Star Block and Squeeze-and-Excitation (SE) Block to form the core unit of feature extraction. The backbone structure of the image feature extraction module realizes multi-level image feature extraction through step-by-step down-sampling and cascaded SE-Star Block modules.

[0014] In the specific implementation process, the SE-Star Block module of each stage works according to the following process: first, the input features are processed by a layer of depth separable convolution, with batch normalization operation. Subsequently, the output features are input to two paths in parallel: the first path passes through convolution (output channel is mlp_ratio*input channel number) and ReLU6 activation function in turn; the second path only includes a layer Convolution. The outputs of the above two paths are fused by element-wise multiplication, and the fused feature map is input to another layer Convolution is used to restore the channel dimension and apply batch normalization operation. Next, the fused features are processed by another layer with depthwise separable convolution, and then input to the SE Block. The output channel attention weight is multiplied with the input features of the SE Block by channel-wise multiplication. Finally, the output features and the initial input of the module are added element-wise through a residual connection to form the final output of the SE-Star Block. As the network depth increases, the spatial dimension of the feature map is sequentially reduced to 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original size, and the number of channels is sequentially increased to d, 2d, 4d and 8d, so that the feature representation with channel attention is extracted at multiple scales.

[0015] In one possible implementation, the spectral feature extraction module adopts a three-stage cascaded feature mapping architecture, and the input is a 6-dimensional non-imaging spectral vector obtained by step 1 data acquisition . The vector is mapped to a high-dimensional feature space with dimension M through layer-by-layer nonlinear transformation, and finally outputs the spectral feature representation .

[0016] Specifically includes:

[0017] The first feature transformation layer is composed of a fully connected layer (input 6-dimensional / output 32-dimensional), a batch normalization layer and a ReLU activation function connected in sequence, which completes the spatial projection and nonlinear activation of the spectral features;

[0018] The second feature transformation layer: expands the feature dimension through a fully connected layer (32→64-dimensional), and realizes deep mapping of the feature space by cooperating with batch normalization and ReLU activation;

[0019] The third feature transformation layer: adopts a fully connected layer (64→M-dimensional) to finally determine the feature representation, and outputs the optimized spectral feature after the same standardization and activation processing .

[0020] In one possible implementation, the low-rank bilinear multi-modal fusion module takes image features and spectral features as input, and outputs joint representation features . The module includes an input feature alignment and projection processing unit, a cross-modal interaction and attention response calculation unit, a multi-attention head feature aggregation and fusion unit, and a cross-modal fusion output generation unit.

[0021] In one possible implementation, the input feature alignment and projection processing unit is implemented as follows:

[0022] Image feature reconstruction and projection: reconstructing and projecting the features of the input image Flattened into a two-dimensional matrix (in ), and through Convolutional dimensionality reduction maps to low-dimensional image features .

[0023]

[0024] in, Represents a non-linear activation function. This represents the image feature projection weight matrix.

[0025] Spectral feature projection: Projecting the input one-dimensional spectral feature vector By embedding the vectors into a semantic space of the same dimension d using linear transformation and activation functions, aligned spectral feature vectors are generated. :

[0026]

[0027] in, This is the spectral projection weight matrix.

[0028] Spatial expansion alignment: will By copying and expanding along the spatial dimension, a spectral feature matrix with the same spatial structure as the image features is obtained. This enables modal alignment.

[0029] In one possible implementation, the cross-modal interaction and attention response calculation unit is specifically implemented as follows:

[0030] Bilinear Interaction: Performing Low-Dimensional Image Features With spectral spread characteristics The element-wise multiplication operation yields the interaction response matrix. :

[0031]

[0032] Attention weight generation: First, the interaction response matrix is... Input to with Each output channel In the convolutional layer, generation A single-channel attention map is generated, and then the attention weight vector is obtained after Softmax normalization. The specific calculation process is as follows:

[0033]

[0034] Wherein, the Softmax operation is to normalize the response values of all spatial positions within each attention channel, so that the attention weights have the property of probability distribution.

[0035] In a possible implementation, the multi-attention head feature aggregation and fusion unit is specifically implemented as follows:

[0036] Multi-attention head feature aggregation: the attention weight vector of each attention head is and the low-dimensional image feature is concatenated to form the input visual feature .

[0037]

[0038] Wherein, represents the feature vector of the input visual feature at the th spatial position (corresponding to the th grid), and the th attention weight vector is the normalized weight value of the th attention weight vector at the th position.

[0039] Subsequently, each is input to an independent feature mapping module (linear mapping + normalization + activation), and the output dimension is . Finally, all attention head outputs are concatenated in the channel dimension to form the multi-head fusion feature .

[0040]

[0041] In a possible implementation, the cross-modal fusion output generation unit is specifically implemented as follows:

[0042] The original spectral feature is mapped to by an extension network (including a linear layer and a hyperbolic tangent activation function) to generate an extended spectral feature ; and the is bilinearly interacted with the multi-head fusion feature , that is, element-wise multiplication, to obtain the final fusion feature .

[0043]

[0044] Classification output: after the final fusion feature is processed by random dropout regularization, it is mapped to the disease category space by a fully connected layer, and the prediction probability of each disease and pest category is output.

[0045]

[0046] In a possible implementation, the proposed lightweight multi-modal recognition model is trained in an end-to-end manner, specifically, a cross-entropy loss function is used to calculate a prediction error, and an AdamW optimizer is used to adaptively adjust parameters to minimize the loss.

[0047] The present application has the following advantages:

[0048] The present application has the following improvements and advantages over the prior art: high precision, early identification and strong robustness: by fusing visible light images and non-imaging spectral data, the present application uses a multi-modal low-rank bilinear pooling mechanism to fully exploit the intuitive representation ability of images and the early lesion identification potential of spectra. Compared with single modal detection methods, the present application can effectively improve the recognition accuracy of rice diseases in complex field environments, and has the ability to identify diseases in the early stage when the disease symptoms are not obvious, which saves time for precise prevention and control. At the same time, by fusing multi-modal features, the present application can effectively address the problems of insufficient generalization ability and high false detection rate of single modal data in complex environments (such as changes in light, occlusion interference), and improve the robustness of the model in actual agricultural production environments.

[0049] Lightweight and resource efficiency: the present application adopts a lightweight neural network structure, including an image feature extraction module using SE-Star Block and a three-level cascaded spectral feature extraction module, as well as a low-rank bilinear multi-modal fusion module. This design significantly reduces the complexity and computational resource consumption of the model, enabling the model to be deployed in intelligent agricultural scenarios with limited hardware resources (such as edge computing devices), achieving low-cost and easy-to-operate characteristics, and overcoming the shortcomings of existing high-spectral technology-based methods that require powerful computing power and high equipment costs.

[0050] Efficient feature fusion mechanism: the low-rank bilinear pooling mechanism is introduced, which fully excavates the complementary information between heterogeneous modalities through input feature alignment and projection processing, cross-modal interaction and attention response calculation, and multi-attention head feature aggregation and fusion units, realizes the deep alignment and fusion expression of cross-modal features, and significantly improves the discrimination ability of multi-modal features. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 The present application is a method for implementing the details of the present application.

[0052] Figure 2 The present application is a method for implementing the details of the present application.

[0053] Figure 3 The present application is a method for implementing the details of the present application. DETAILED DESCRIPTION

[0054] The technical solutions of the present application are further described below in combination with the drawings and examples.

[0055] The lightweight rice disease identification method based on multi-modal low-rank bilinear pooling includes the following steps:

[0056] Step 1: Multi-modal pest and disease data acquisition and dataset construction.

[0057] Multi-modal rice pest and disease atlas data with consistent field of view are collected, including RGB image data and six-band non-imaging spectral reflectance data.

[0058] Among them, the image data is collected by a conventional visible light imaging device, which has good texture and structure representation ability; the spectral data is synchronously collected by a non-imaging sensor, covering the following six center bands: 530nm, 570nm, 680nm, 700nm, 740nm and 780nm. The two modal data are consistent in space field of view, which is convenient for subsequent fusion processing.

[0059] In order to enhance the representativeness and adaptability of the data, the pest and disease atlas data is collected in four typical rice planting areas in Zhejiang Province, China. The collection period is from 2023 to 2024, ensuring coverage of multiple growth seasons and disease occurrence stages. The data collection time is limited to 10:00 to 15:00 on sunny days to reduce light interference; the collection distance is uniformly controlled at about 50cm from the rice canopy.

[0060] The pest and disease categories cover four typical and significantly affecting rice production pest and disease types, namely Chilo suppressalis, rice blast, rice leaf roller and bacterial leaf blight. In order to achieve fine identification of disease degree, each pest and disease is divided into two grades of mild and severe, combined with healthy samples, a total of nine recognition labels are set, labeled as Label 1 to Label 9, the specific corresponding relationship is as follows:

[0061] Label 1 and Label 2 represent mild and severe Chilo suppressalis, respectively;

[0062] Label 3 and Label 4 represent mild and severe rice blast, respectively;

[0063] Label 5 and Label 6 represent mild and severe rice leaf roller, respectively;

[0064] Label 7 and Label 8 represent mild and severe bacterial leaf blight, respectively;

[0065] Label 9 represents healthy plants.

[0066] Data preprocessing:

[0067] The collected rice disease and pest atlas data is preprocessed. For image data, first calculate the mean and variance of each channel and perform normalization processing, then adjust the image to 224x224 pixels. For the corresponding non-imaging spectral data, only normalization processing is performed to keep the numerical scale consistent. This embodiment contains a total of 3945 groups of atlas samples, each group of samples consists of an RGB image and its corresponding 6-dimensional non-imaging spectral vector. After preprocessing, the total samples are divided into training set, validation set and test set in the ratio of 6:2:2, respectively, for model training, performance verification and final evaluation.

[0068] Step 2: Construct a lightweight multi-modal recognition model.

[0069] The model includes a double-branch structure composed of an image feature extraction module and a spectral feature extraction module, and a low-rank bilinear multi-modal fusion module. Through the double-branch structure, image features and spectral features are obtained respectively, and finally through the low-rank bilinear multi-modal fusion module, the image and spectral features are fully fused, thereby enhancing the recognition ability of the model.

[0070] Image feature extraction module:

[0071] The image feature extraction module adopts a classic four-stage hierarchical structure. Each stage contains a convolutional layer (step size 2) for feature map size downsampling, and cascades multiple SE-Star Block modules, which effectively extract multi-scale features of RGB images through this hierarchical design.

[0072] In this embodiment, the SE-Star Block configuration of the four stages is as follows:

[0073] The first stage contains 1 SE-Star Block.

[0074] The second stage contains 2 SE-Star Blocks.

[0075] The third stage contains 6 SE-Star Blocks.

[0076] The fourth stage contains 2 SE-Star Blocks.

[0077] It should be noted that the SE-Star Block is a lightweight improved module that combines the advantages of Star Block and Squeeze-and-Excitation (SE) Block. The number and structure configuration can be flexibly adjusted and extended according to actual needs, and is not limited to the specific parameters in the embodiment. The backbone structure of the image feature extraction module realizes multi-level feature extraction of the rice disease RGB image through stepwise downsampling and cascading SE-Star Block modules, and forms image features for subsequent fusion .

[0078] In the specific implementation process, the SE-Star Block module of each stage works according to the following process: first, the input features are processed by a layer of depth separable convolution, the convolution kernel size is , the stride is 1, the padding is 3, the group number is equal to the input channel number, and the batch normalization operation is attached. Subsequently, the output features are input into two paths in parallel: the first path sequentially passes through convolution (the output channel is mlp_ratio x input channel number, where mlp_ratio is set to 4 for example) and ReLU6 activation function; the second path only includes a layer of convolution with the same output channel number. The outputs of the above two paths are fused by element-wise multiplication (i.e. star operation "*"), and the fused feature map is input into another layer convolution to restore the channel dimension and apply batch normalization operation. Next, the fused features are processed by a layer of depth separable convolution with the same parameters as described above, and then input into the SE Block. The SE Block first compresses each channel to by global average pooling operation, and then compresses the channel number to se_ratio times and restores it to the original channel number by two layers of convolution, with ReLU6 and Sigmoid activation functions, and finally the output channel attention weight is fused with the main branch feature by channel-wise multiplication. Finally, the output features and the initial input of the module are added element by element through residual connection to form the final output of the module. It should be noted that the Convolution is structurally equivalent to a fully connected layer (FC) because it essentially performs a linear transformation operation on the channel dimension independently at each spatial location. Through the above design, the SE-Star Block of each stage can both fuse multi-branch features through the Star Block and enhance key feature expression using the channel attention mechanism of the SE Block. As the network depth increases, the spatial dimension of the feature map is sequentially reduced to 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original size, and the number of channels is sequentially increased to d (for example, set to 64), 2d, 4d, and 8d, thereby extracting channel-attention features at multiple scales.

[0079] The spectral feature extraction module includes:

[0080] The module extracts spectral features through progressive feature enhancement , and provides high-discriminative inputs for the subsequent modal fusion module. The spectral feature extraction module adopts a three-stage cascaded feature mapping architecture, and the input is a 6-dimensional non-imaging spectral vector obtained through step 1 data collection. The vector is gradually mapped to a high-dimensional feature space with a dimension of 96 through layer-by-layer nonlinear transformation, and finally outputs a spectral feature representation .

[0081] Specifically includes:

[0082] The first feature transformation layer is composed of a fully connected layer (input 6-dimensional / output 32-dimensional), a batch normalization layer, and a ReLU activation function connected in sequence, which completes the spatial projection and nonlinear activation of the spectral features;

[0083] The second feature transformation layer: expands the feature dimension through a fully connected layer (32→64-dimensional), cooperates with batch normalization and ReLU activation, and realizes deep mapping of the feature space;

[0084] The third feature transformation layer: adopts a fully connected layer (64→96-dimensional) to finally determine the feature representation, and outputs the optimized spectral feature after the same standardization and activation processing.

[0085] Low-rank bilinear multi-modal fusion module:

[0086] Through the low-rank bilinear multi-modal fusion module, efficient fusion between RGB image features and non-imaging spectral features is realized. The module introduces a multi-level attention mechanism and a bilinear interaction structure to fully exploit the complementary information between heterogeneous modalities, realize deep alignment and fusion expression of cross-modal features, and significantly improve the discriminability of multi-modal features. The low-rank bilinear multi-modal fusion module takes image features and spectral features For input, output joint representation features , for supporting subsequent rice pest and disease identification tasks, enhancing the robustness and identification accuracy of the model. The module includes an input feature alignment and projection processing unit, a cross-modal interaction and attention response calculation unit, a multi-attention head feature aggregation and fusion unit, and a cross-modal fusion output generation unit:

[0087] The input feature alignment and projection processing unit is implemented as follows:

[0088] Image feature remodeling and projection: the input image feature is flattened into a two-dimensional matrix , and is mapped to a low-dimensional image feature by convolution dimension reduction ;

[0089]

[0090] wherein, represents a nonlinear activation function, specifically a hyperbolic tangent function Tanh, represents an image feature projection weight matrix.

[0091] Spectral feature projection: the input one-dimensional spectral feature vector is embedded into the same semantic space of dimension d by linear transformation and activation function, generating the aligned spectral feature vector :

[0092]

[0093] wherein, is a spectral projection weight matrix. Spatial expansion alignment: the is expanded along the spatial dimension, and the spectral expansion feature is generated by copying to realize the position alignment of the image feature in the spatial dimension, and to realize the modal unification.

[0094] The cross-modal interaction and attention response calculation unit is implemented as follows:

[0095] Bilinear interaction: perform element-wise multiplication operation of low-dimensional image feature and spectral expansion feature , to get the interaction response matrix:

[0096]

[0097] Attention weight generation: first, input the interaction response matrix to a convolutional layer with output channels, to generate convolutional layer with a single-channel attention map, and then a normalized attention weight vector is obtained after Softmax normalization The specific calculation process is as follows:

[0098] The Softmax operation is to normalize the response values of all spatial positions inside each attention channel, so that the attention weight has the property of probability distribution.

[0099] The multi-attention head feature aggregation and fusion unit is specifically implemented as follows:

[0100] Multi-attention head feature aggregation: the attention weight vector of each attention head is and the low-dimensional image feature is weighted and summed to generate a visual vector under the guidance of attention The calculation method is as follows:

[0101]

[0102] wherein, represents the feature vector of the input visual feature at the th spatial position (corresponding to the th grid), is the normalized weight value of the th attention weight vector at the th position.

[0103] Subsequently, each is input to an independent feature mapping module (linear mapping + normalization + activation), and the output dimension is . Finally, all attention head outputs are spliced in the channel dimension to form a multi-head fusion feature .

[0104]

[0105] The cross-modal fusion output generation unit is specifically implemented as follows:

[0106] The original spectral feature is mapped to by an extension network (including a linear layer and a hyperbolic tangent activation function), to generate an extended spectral feature ; and is bilinearly interacted with the multi-head fusion feature , that is, element-wise multiplication, to obtain the final fusion feature .

[0107]

[0108] ​Classification output: final fused feature After the regularization by random inactivation, the prediction probability of each disease and pest category is output by mapping to the disease category space through the fully connected layer.

[0109]

[0110] Step 3: model training.

[0111] The lightweight multi-modal recognition model constructed in step two is trained end-to-end using the training set. During the training process, the cross-entropy loss function is used to measure the difference between the model prediction results and the true labels, and the AdamW optimizer is used to update the model parameters to minimize the loss function.

[0112] In this embodiment, the initial learning rate of the AdamW optimizer is set to 1e-3, the weight decay coefficient is set to 0.1, and other optimizer parameters are set to default, such as momentum factor betas=(0.9, 0.999) and numerical stability term eps=1e-8. During the training process, the model performance is continuously monitored on the validation set. If the validation set accuracy does not improve for several consecutive rounds, the early stopping mechanism is triggered to prevent overfitting of the model.

[0113] The model training parameters include but are not limited to the initial learning rate, the batch size (set to 32 in this embodiment), and the number of training rounds (the maximum number of rounds is set to 200). The above parameters can be adjusted according to the specific task requirements to improve the model performance and training efficiency.

[0114] Step 4: model evaluation and experimental results.

[0115] To verify the effectiveness of the method, the performance evaluation was conducted on the constructed rice disease and pest atlas dataset. The results show that the multi-modal fusion model proposed in this invention achieved an identification accuracy of 95.22%, an accuracy of 96.00%, a recall rate of 95.27%, and an F1 score of 95.59% on the test set. Compared with the identification accuracy of the image branch (90.55%) and the identification accuracy of the spectrum branch (85.49%), the fusion model of this invention achieved significant improvement in accuracy, fully verifying the effectiveness of the multi-modal collaborative modeling strategy. To further demonstrate the performance of the model in multi-classification tasks, the confusion matrix (see Figure 3 ) was drawn, which clearly represents the discrimination effect of the model on nine types of rice disease states. The model parameter amount is only 3.75M, which ensures the recognition accuracy while maintaining good lightweight characteristics, and has the feasibility of deployment on resource-limited devices, with high practicality and promotion value.

[0116] It is to be noted that, as used in this document, the term "thick" refers to a thickness of a layer of material, and the term "thin" refers to a thickness of a layer of material that is less than the thickness of the layer of material referred to as "thick". It is to be understood that the terms "first" and "second" are used herein only to distinguish one element from another, and do not imply any physical or logical relationship or order of one element to another. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0117] Each of the embodiments in the present specification is described in a related manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between the embodiments can be understood by referring to each other.

[0118] The above only describes the preferred embodiments of the present application, and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A lightweight rice disease identification method based on multimodal low-rank bilinear pooling, characterized in that, Includes the following steps: Step 1: Data collection and dataset construction; RGB images of pests and diseases in the rice canopy and non-imaging spectral data in six bands (530nm, 570nm, 680nm, 700nm, 740nm, and 780nm) were collected to obtain a 6-dimensional non-imaging spectral vector. Registration and preprocessing were performed to construct a multimodal dataset for training and evaluation. Step 2: Construct a lightweight multimodal recognition model; The model includes a dual-branch structure consisting of an image feature extraction module and a spectral feature extraction module, as well as a low-rank bilinear multimodal fusion module; Step 3: Obtain image features and spectral features respectively through a dual-branch structure; Step 4: By using the low-rank bilinear multimodal fusion module, image and spectral features are fully fused to obtain joint representation features, thereby realizing the identification of rice diseases.

2. The lightweight rice disease identification method based on multimodal low-rank bilinear pooling according to claim 1, characterized in that, The image feature extraction module adopts a four-stage hierarchical structure; each stage includes a downsampling layer and an SE-StarBlock module. This hierarchical design effectively extracts features from RGB images. Among them, the SE-Star Block combines the advantages of the Star Block and the SE Block to form the core unit of feature extraction; the backbone structure of the image feature extraction module realizes multi-level image feature extraction through progressive downsampling and cascaded SE-Star Block modules. In the specific implementation process, the SE-Star Block module at each stage operates according to the following process: First, the input features are processed by a depthwise separable convolution layer, with batch normalization; then, the output features are input in parallel to two paths: the first path sequentially passes through... Convolution and ReLU6 activation function; the second path only includes one layer. Convolution; the outputs of the two paths mentioned above are fused through element-wise multiplication, and the fused feature map is input to another layer. Convolution is performed to restore the channel dimension and batch normalization is applied. Next, the fused features are processed by a depthwise separable convolution and then input into the SE Block. The output channel attention weights are fused with the input features of the SE Block through channel-wise multiplication. Finally, the output features are added element-wise to the initial input of the module through residual connections to form the final output of the SE-Star Block. As the network depth increases, the spatial dimension of the feature map is reduced to 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original size in stages, while the number of channels increases to d, 2d, 4d and 8d in stages, thereby extracting feature representations with channel attention at multiple scales.

3. The lightweight rice disease identification method based on multimodal low-rank bilinear pooling according to claim 1, characterized in that, The spectral feature extraction module adopts a three-level cascaded feature mapping architecture, and its input is the 6-dimensional non-imaging spectral vector obtained from the data acquisition in step 1. The vector is gradually mapped to a high-dimensional feature space of dimension M through a layer-by-layer nonlinear transformation, and finally outputs a spectral feature representation. ; Specifically, it includes: The first feature transformation layer consists of a fully connected layer, a batch normalization layer, and a ReLU activation function connected in sequence, which completes the spatial projection and nonlinear activation of spectral features. The second feature transformation layer expands the feature dimension through a fully connected layer, and, in conjunction with batch normalization and ReLU activation, achieves a deep mapping of the feature space. The third feature transformation layer: A fully connected layer is used to finally determine the feature representation. After the same normalization and activation processing, the optimized spectral features are output. .

4. The lightweight rice disease identification method based on multimodal low-rank bilinear pooling according to claim 1, characterized in that, The low-rank bilinear multimodal fusion module uses image features With spectral characteristics The input and output are joint representation features. This module includes an input feature alignment and projection processing unit, a cross-modal interaction and attention response calculation unit, a multi-attention head feature aggregation and fusion unit, and a cross-modal fusion output generation unit.

5. The lightweight rice disease identification method based on multimodal low-rank bilinear pooling according to claim 4, characterized in that, The input feature alignment and projection processing unit is implemented as follows: Image feature reconstruction and projection: reconstructing and projecting the features of the input image Flattened into a two-dimensional matrix ,in and through Convolutional dimensionality reduction maps to low-dimensional image features ; in, Represents a non-linear activation function. Represents the image feature projection weight matrix; Spectral feature projection: Projecting the input one-dimensional spectral feature vector By embedding the vectors into a semantic space of the same dimension d using linear transformation and activation functions, aligned spectral feature vectors are generated. : in, This is the spectral projection weight matrix; Spatial expansion alignment: will By copying and expanding along the spatial dimension, a spectral feature matrix with the same spatial structure as the image features is obtained. This enables modal alignment.

6. The lightweight rice disease identification method based on multimodal low-rank bilinear pooling according to claim 5, characterized in that, The cross-modal interaction and attention response calculation unit is specifically implemented as follows: Bilinear Interaction: Executing Low-Dimensional Image Features With spectral spread characteristics The element-wise multiplication operation yields the interaction response matrix. : Attention weight generation: First, the interaction response matrix is... Input to with Each output channel In the convolutional layer, generation A single-channel attention map is generated, and then the attention weight vector is obtained after Softmax normalization. The specific calculation process is as follows: The Softmax operation normalizes the response values ​​at all spatial locations within each attention channel, giving the attention weights a probability distribution property.

7. The lightweight rice disease identification method based on multimodal low-rank bilinear pooling according to claim 6, characterized in that, The multi-attention head feature aggregation and fusion unit is specifically implemented as follows: Multi-attention head feature aggregation: This involves combining the attention weight vectors of each attention head. With low-dimensional image features Attention-guided visual vectors are generated through weighted summation. , The calculation method is as follows: in, Indicates the input visual features in the th... Feature vectors of spatial locations ( correspond (grid) For the first The attention weight vector at the th ... Normalized weight values ​​for position; Subsequently, each The input is fed into a separate feature mapping module, and the output dimension is... Finally, by splicing together the outputs of all attention heads along the channel dimension, a multi-head fusion feature is formed. ; 。 8. The lightweight rice disease identification method based on multimodal low-rank bilinear pooling according to claim 7, characterized in that, The cross-modal fusion output generation unit is specifically implemented as follows: Original spectral features By extending the network, mapping to Generate extended spectral features ;Will Multi-head fusion features Perform bilinear interaction, i.e., element-wise multiplication, to obtain the final fused features. ; Classification output: Final fused features After random inactivation regularization, the data is mapped to the disease category space through a fully connected layer, and the predicted probability of each disease and pest category is output. 。 9. The lightweight rice disease identification method based on multimodal low-rank bilinear pooling according to claim 1, characterized in that, The proposed lightweight multimodal recognition model is trained in an end-to-end manner. Specifically, the prediction error is calculated using the cross-entropy loss function, and the parameters are adaptively adjusted using the AdamW optimizer to minimize the loss.

Images