Wheat early-stage stripe rust detection method and system

Abstract

The invention provides a wheat early-stage stripe rust detection method and system, and the method comprises the steps: inputting time dimension embedded multi-modal data, pre-training a spectrum reconstruction and time sequence comparison learning model, and mining universal feature modes of wheat stripe rust in different development stages and different spectral bands; and transferring the pre-trained weight to a multi-modal encoder of a multi-modal multi-scale target detection model, extracting interactive fusion features of different scales, inputting the interactive fusion features into a multi-scale decoder, comprehensively utilizing global context information and local detail information, and meanwhile, fully utilizing the features of different scales through a cross attention mechanism so as to obtain a multi-scale multi-scale target detection model. Outputting a stripe rust detection result; finely adjusting parameters of the multi-scale decoder by using data with labels in the early-stage stripe rust data to obtain a trained multi-modal multi-scale target detection model; and performing early stripe rust detection on wheat to be detected by using the trained multi-modal multi-scale target detection model. The kit can be used for detecting the early wheat stripe rust.

Description

Technical Field

[0001] The present invention relates to the technical field of early stripe rust detection of wheat, in particular to a method and system for early stripe rust detection of wheat. Background Art

[0002] As a fungal disease that seriously affects wheat production, stripe rust poses a serious threat to global food security. The process of wheat being infected by stripe rust is divided into an invasion period, an incubation period, and a disease onset period in terms of time. During the invasion period, the pathogen has just invaded the leaves and has not yet affected the health and appearance of the plant. During the incubation period, the internal change mechanism and disease symptoms of wheat are extremely complex, making it very difficult to distinguish stripe rust with the naked eye. When the stripe rust lesions are visible to the human eye, it is already the disease onset period, at which time stripe rust has caused irreversible damage to the growth of wheat and affected the final yield of wheat. Therefore, how to identify wheat stripe rust during the incubation period is the core problem of early stripe rust detection.

[0003] In recent years, with the continuous development of deep learning technology, significant progress has been made in the research on early stripe rust detection of wheat. However, there are still many problems in traditional deep learning-based detection in actual operation. First, the information carried by a single-modal data source (such as the visible light band) is insufficient and cannot provide complete material composition information, which limits the ability of images to characterize the characteristics of stripe rust. Second, the incubation period of wheat stripe rust is relatively short and quickly turns into the disease onset period. Therefore, there are few sample data of early (incubation period) stripe rust, but there are many sample data in the late (disease onset period), resulting in a scarcity of high-quality labeled early stripe rust, and it is difficult for the model to learn the true characteristics of early stripe rust from a small amount of labeled data. Finally, the symptoms of stripe rust in the early stage are relatively hidden, and the spatial characteristics of the lesion area are not obvious. Traditional object detection models are difficult to capture the unobvious spatial characteristics. Summary of the Invention

[0004] In order to solve the above technical problems existing in the prior art, the present invention provides a method for early stripe rust detection of wheat, and the technical solution is as follows:

[0005] On the one hand, a method for early stripe rust detection of wheat is provided, and the method includes:

[0006] S1. Obtain and preprocess multi-modal image data and corresponding multi-modal time data of early and late wheat, comprehensively use the early data to make detection box labels, and divide them into a training set and a validation set;

[0007] S2, inputting the early multimodal image data, late multimodal image data and corresponding multimodal time data in the training set into the time feature fusion module, fully extracting and fusing the multimodal data features of different time periods, obtaining a richer and more comprehensive feature representation of stripe rust than single modal data, and obtaining multimodal data embedded in the time dimension;

[0008] S3, input the multimodal data embedded in the time dimension and pre-train the spectral reconstruction and time series contrast learning model, dig out the general and common feature patterns of wheat stripe rust at different development stages and in different spectral bands, the spectral reconstruction and time series contrast learning model includes two parallel branches, one branch randomly covers part of the image information and performs spectral reconstruction training, by letting the feature reconstructor in the model learn to restore the covered information, forcing the feature extractor in the model to learn the general disease space features; the other branch constructs a comparison sample pair after data enhancement operation, performs time series contrast training, and uses contrast loss to force the feature extractor in the model to learn to map data in the same time period to a similar feature space, and data in different time periods to a farther feature space, and uses time information to force the model to distinguish the stripe rust features at different time points, prompting the feature extractor to more keenly capture the early features of stripe rust;

[0009] S4, transferring the weights of the pre-trained feature extractor to the multimodal encoder of the multimodal multi-scale target detection model, the multimodal encoder automatically adjusts the weights of different modal features in the feature space, performs more sophisticated multimodal feature extraction and fusion, extracts interactive fusion features of different scales, and inputs them into the multiscale decoder of the multimodal multi-scale target detection model. The multiscale decoder introduces high-level features learned at low resolution into high-resolution features through skip connections, comprehensively utilizes global context information and local detail information, and fully utilizes features of different scales through a cross-attention mechanism to output the detection results of stripe rust;

[0010] S5. Fine-tune the multi-scale decoder parameters of the multi-modal multi-scale target detection model using the labeled data in the early stripe rust data to obtain a trained multi-modal multi-scale target detection model;

[0011] S6. Use the trained multi-modal and multi-scale target detection model to perform early stripe rust detection on the wheat to be detected.

[0012] Optionally, the S1 specifically includes:

[0013] The drone was equipped with red light, near infrared light, visible light and polarized light cameras to shoot the entire wheat field and obtain the red light image data of wheat stripe rust for several consecutive days in the early stage. 1 , near infrared image data D 2, visible light image data D 3 , polarized light image data D 4 and early time data T 1 , when the stripe rust enters the late disease course stage with obvious characteristics, the entire wheat field is photographed again by the drone to obtain the red light image data W of the stripe rust in the late stage for multiple consecutive days 1 , near-infrared light image data W 2 , visible light image data W 3 , polarized light image data W 4 and late time data T 2 ;

[0014] Perform spatial position registration on the early stripe rust image data D 1 , D 2 , D 3 , D 4 and the late stripe rust image data W 1 , W 2 , W 3 , W 4 respectively. Through geometric correction, ensure that the resolutions, projections, and coordinate systems of all modal images are consistent, eliminate the errors caused by the shooting angle, and provide consistent spatial information for subsequent data annotation and model training;

[0015] By observing the manifestations of the early stripe rust symptoms at different modalities and different times, combined with the visible light, polarized light, near-infrared light, and red light image data, make detection box labels for the early multi-modal image data and divide them into a training set and a validation set.

[0016] Optionally, the S2 specifically includes:

[0017] Input the early data (D 1 , D 2 , D 3 , D 4 , T 1 ) and the late data (W 1 , W 2 , W 3 , W 4 , T 2 ) into the time feature fusion module to obtain the early image data (D′ 1 , D′ 2 , D′ 3 , D′ 4 ) and the late image data (W′ 1 , W′ 2 , W′ 3 , W′ 4) Among them, the time feature fusion module performs 3x3 convolution repeated 3 times on the multi-modal image data, followed by a normalization layer, ReLU activation function, 3x3 convolution, and normalization layer connected by a 1x1 convolution residual. The operation result is subjected to cross-attention operation with the multi-spectral time data after linear mapping. The operation result passes through the normalization layer with residual connection repeated 2 times, Dropout and normalization layer, feed-forward network with residual connection, normalization layer, and feed-forward network, and outputs the multi-modal data embedded in the time dimension.

[0018] Optionally, the spectral reconstruction training in S3 specifically includes:

[0019] Concatenate the input early and late multi-modal data embedded in the time dimension along the channel dimension, and cut the concatenated image into image patches with a size of 16x16 pixels for each spectral channel;

[0020] Then randomly mask 50% of the image patches, add position encoding information, and input them into the feature extractor to extract the spectral features represented by the unmasked image patches;

[0021] Then input the extracted spectral features into the feature reconstructor, use the spectral features extracted by the feature extractor to reconstruct the masked image patches. The spectral reconstruction loss adopts the mean square error MSE, calculate the difference between the pixels of the image patches reconstructed by the feature reconstructor and the input image patches, and enhance the model's perception ability of the late stripe rust spectral information by using the spectral reconstruction loss to supervise the model learning, obtaining a general feature extractor that can process multiple modal spectral inputs;

[0022] The temporal contrast training in S3 specifically includes:

[0023] Perform two random data augmentation operations on the early and late data respectively to obtain augmented data 1 (D′ 1.1 , D′ 2.1 , D′ 3.1 , D′ 4.1 , W′ 1.1 , W′ 2.1 , W′ 3.1 , W′ 4.1 ) and augmented data 2 (D′ 1.2 , D′ 2.2 , D′ 3.2 , D 4.2 , W 1.2 , W 2.2 , W 3.2 , W 4.2 );

[0024] Then, the 16 enhanced data in two parts are combined in pairs to construct 120 non-repeating comparison sample pairs. After the comparison sample pairs are extracted with features by the feature extractor, they are processed by a linear layer, an activation function, and a normalization layer to obtain 120 comparison feature pairs L 1 , L 2 , L 3 ,......, L 120 , calculate the similarity between each comparison sample pair through cosine, and then calculate the temporal contrast loss through the normalized cross-entropy loss function N-Xent. The calculation formula is as follows:

[0025]

[0026]

[0027] Through temporal contrast learning training, the model can understand the variation law of the spectral characteristics of stripe rust over time;

[0028] A joint loss function composed of mean square error and normalized cross-entropy loss function is used to backpropagate and optimize the parameters of the feature extractor and the feature reconstructor, so that the model can capture spatio-temporal joint features, while considering the disease manifestations in space and the evolution process over time.

[0029] Optionally, the feature extractor includes 12 layers, each layer consists of a linear layer, a normalization layer connected by a 3×3 convolutional residual, and a self-attention layer, plus a normalization layer connected by a 3×3 convolutional residual and a multi-layer shared perceptron;

[0030] On the one hand, for the feature reconstructor, the input image is processed by the difference between max pooling and average pooling to screen out the part with high feature values in the input image. After preliminary feature extraction by a 3×3 convolution, it is input into the activation function to obtain the weight matrix of the pixels in different regions of the image. The input image is multiplied by the weight matrix after passing through a 3×3 convolution, focusing on the local regions with important information in the image and learning local features;

[0031] On the other hand, the input image dynamically adjusts the attention degree of each input position to other positions through layer normalization, self-attention mechanism, and 3×3 convolution, integrates global information to form a more accurate representation, and learns global features;

[0032] After splicing the learned local features and global features, they are input into a multi-layer perceptron to output the reconstructed feature image.

[0033] Optionally, the multi-modal encoder is composed of three multi-modal coding networks stacked repeatedly. Each multi-modal coding network consists of four parallel sub-feature extractors with the same weights and a multi-modal fusion network. The sub-feature extractors E of the three multi-modal coding networks1 , E 2 and E 3 load the weights of the 1st - 4th layers, 5th - 8th layers, and 9th - 12th layers of the pre - trained feature extractor respectively, for extracting multi - modal features and inputting them into the multi - modal fusion network;

[0034] The multi - modal fusion network consists of interactive modal fusion modules. For the input features Figure 1 and features Figure 2 after linear mapping, query vectors Q 1 , key vectors K 1 , value vectors V 1 and query vectors Q 2 , key vectors K 2 , value vectors V 2 are obtained respectively. The query vector Q Figure 1 of feature 1 multiplies the transpose of the key vector K Figure 2 of feature 2 to obtain a weight matrix. After sigmoid processing of the weight matrix, it multiplies the value vector V Figure 2 of feature 2 to obtain a weighted matrix of the attention degrees of different parts of feature Figure 1 to feature Figure 2 ; The query vector Q Figure 2 of feature 1 multiplies the transpose of the key vector K Figure 1 of feature 1 , and after sigmoid processing, it multiplies the value vector V Figure 1 of feature 1 to obtain a weighted matrix of the attention degrees of different parts of feature Figure 2 to feature Figure 1 ; The two weighted matrices and feature Figure 1 and feature Figure 2 are concatenated by channels and then a 1×1 convolution operation is performed to obtain the fused feature;

[0035] In multi - modal fusion network 1 and multi - modal fusion network 2, first, the red light and near - infrared light are fused through the interactive modal fusion module, the obtained fusion result is fused with the visible light through the interactive modal fusion module, and then the new fusion result is fused with the polarized light through the interactive modal fusion module again. One side of the fusion result is output as the interactive fusion feature M, and the other side is respectively concatenated with the original red light, near - infrared light, visible light, and polarized light feature maps in the channel dimension, and then input into the deep multi - modal encoding network after 2 - fold downsampling as input features. Multi - modal fusion network 3 only outputs the interactive fusion feature M and no longer outputs the input of the deep network. Three different spatial - scale interactive fusion features M are obtained through three groups of multi - modal encoding networks1 , M 2 , M 3 .

[0036] Optionally, the input of the multi-scale decoder is the interactive fusion feature M 1 , M 2 , M 3 ;

[0037] M 3 Undergoes a 3×3 convolution to obtain the interactive fusion feature M' 3 ;

[0038] M 3 After being upsampled by a factor of 2, it is concatenated with M 2 and a 1×1 convolution operation is performed to obtain M 2,3 , and then undergoes a 3×3 convolution to obtain the interactive fusion feature M' 2 ;

[0039] M 2 After being upsampled by a factor of 2, it is concatenated with M 1 and a 1×1 convolution operation is performed to obtain M 1,2 , and then it is concatenated with M 2,3 After being upsampled by a factor of 2, and a 1×1 convolution operation is performed to obtain M 1,2,3 , and then a 3×3 convolution operation is performed to obtain the interactive fusion feature M' 1 ;

[0040] The interactive fusion features M' of different scales 1 , M' 2 , M' 3 After being flattened, are input into a linear mapping layer for mapping to form feature Tokens. The feature Tokens obtained from the feature maps of different scales are concatenated according to the Token quantity dimension. The result is used as the value vector, and the concatenated feature Tokens are added with positional encoding and used as the key vector. At the same time, N target query vectors are randomly initialized according to the Gaussian distribution, where N is the upper limit of the prediction instance targets that the model can give;

[0041] The key, value, and target query vectors are subjected to cross-attention operations to obtain the feature attention matrix. After the feature attention matrix is layer-normalized, it is input into a self-attention layer with residual connections and layer-normalized. Then, the operation result is input into a feed-forward neural network with residual connections and layer-normalized again, thus completing one query process. After repeating the query process 6 times, the predicted query feature is obtained. The predicted query feature is input into two parallel feed-forward neural networks and mapped to the detection box coordinate prediction value and the confidence prediction value through linear layers respectively.

[0042] On the other hand, a wheat early stripe rust detection system is provided, the system comprising:

[0043] A preprocessing module is obtained to obtain and preprocess the multimodal image data of early and late wheat and the corresponding multimodal time data, comprehensively utilize the early data to make detection frame labels, and divide them into training sets and validation sets;

[0044] The temporal feature fusion module is used to input the early multimodal image data, late multimodal image data and corresponding multimodal time data in the training set into the temporal feature fusion module, fully extract and fuse the multimodal data features of different time periods, obtain a richer and more comprehensive feature representation of stripe rust than single modal data, and obtain multimodal data embedded in the time dimension;

[0045] A pre-training module is used to input the multimodal data embedded in the time dimension and pre-train a spectral reconstruction and time series contrast learning model to mine general and common feature patterns of wheat stripe rust at different development stages and in different spectral bands. The spectral reconstruction and time series contrast learning model includes two parallel branches. One branch randomly masks part of the image information and then performs spectral reconstruction training. By allowing the feature reconstructor in the model to learn to restore the masked information, the feature extractor in the model is forced to learn the general disease space features. The other branch constructs a comparison sample pair after data enhancement operation and performs time series contrast training. By using contrast loss, the feature extractor in the model is forced to learn to map data in the same time period to a similar feature space, and data in different time periods to a farther feature space. The time information is used to force the model to distinguish the stripe rust features at different time points, so that the feature extractor can more keenly capture the early features of the stripe rust.

[0046] The target detection module is used to transfer the weights of the pre-trained feature extractor to the multimodal encoder of the multimodal multi-scale target detection model. The multimodal encoder automatically adjusts the weights of different modal features in the feature space, performs more sophisticated multimodal feature extraction and fusion, extracts interactive fusion features of different scales, and inputs them into the multiscale decoder of the multimodal multi-scale target detection model. The multiscale decoder introduces high-level features learned at low resolution into high-resolution features through skip connections, comprehensively utilizes global context information and local detail information, and fully utilizes features of different scales through a cross-attention mechanism to output the detection results of stripe rust;

[0047] A fine-tuning module, used to fine-tune the multi-scale decoder parameters of the multi-modal multi-scale target detection model using labeled data in the early stripe rust data to obtain a trained multi-modal multi-scale target detection model;

[0048] A detection module, which is used to perform early stripe rust detection on the wheat to be detected by using the trained multi-modal multi-scale object detection model.

[0049] On the other hand, an electronic device is provided. The electronic device includes a processor and a memory. At least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor to implement the above-mentioned wheat early stripe rust detection method.

[0050] On the other hand, a computer-readable storage medium is provided. At least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the above-mentioned wheat early stripe rust detection method.

[0051] The beneficial effects brought by the technical solution provided by the present invention at least include:

[0052] 1. The present invention uses remote sensing technology to obtain data of four modalities: red light, near-infrared light, visible light, and polarized light, and designs a temporal feature fusion module to fuse the data features of the four modalities and temporal features, improving the spectral information density of the image, which can alleviate the problem of insufficient information in a single data source and provide high-quality data for the model to extract features.

[0053] 2. The present invention designs a spectral reconstruction and temporal contrast learning model, which can make full use of early data and late data incorporating temporal information to train the model. The model parameters learned by the model can be transferred to the detection model. On the one hand, it solves the problem of insufficient training of the detection model caused by the imbalance of early and late sample data, and on the other hand, it solves the problem of poor generalization ability of the detection model.

[0054] 3. The multi-modal encoder designed by the present invention can automatically adjust the weights of different modal features in the feature space, thereby achieving more refined feature fusion. By fusing features of different modalities, the model can obtain more comprehensive plant health information, thereby improving the sensitivity and accuracy of the model to weak lesions of early stripe rust and enhancing the comprehensive perception ability of the plant health status.

[0055] 4. The multi-scale decoder designed by the present invention introduces high-level features learned at low resolution into high-resolution features through skip connections, which can comprehensively utilize global context information and local detail information. At the same time, through the cross-attention mechanism, it makes full use of features of different scales, solving the problem that early stripe rust is difficult to detect due to unclear spatial features. Description of the Drawings

[0056] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0057] Figure 1 It is a flowchart of a method for detecting early stripe rust of wheat provided by an embodiment of the present invention;

[0058] Figure 2 It is a general block diagram of a method for detecting early stripe rust of wheat provided by an embodiment of the present invention;

[0059] Figure 3 It is a block diagram of the structure of a time feature fusion module provided by an embodiment of the present invention;

[0060] Figure 4 It is a block diagram of the structure of a spectral reconstruction and temporal contrast learning model provided by an embodiment of the present invention;

[0061] Figure 5 It is a block diagram of the structure of a feature extractor provided by an embodiment of the present invention;

[0062] Figure 6 It is a block diagram of the structure of a feature reconstructor provided by an embodiment of the present invention;

[0063] Figure 7 It is a block diagram of the structure of a multi-modal encoder provided by an embodiment of the present invention;

[0064] Figure 8 It is a block diagram of the structure of a multi-modal fusion network provided by an embodiment of the present invention;

[0065] Figure 9 It is a block diagram of the structure of an interactive modal fusion module provided by an embodiment of the present invention;

[0066] Figure 10 It is a block diagram of the structure of a multi-scale decoder provided by an embodiment of the present invention;

[0067] Figure 11 It is a block diagram of a system for detecting early stripe rust of wheat provided by an embodiment of the present invention;

[0068] Figure 12 It is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0069] To make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the following will be described in detail with reference to the accompanying drawings and specific embodiments.

[0070] An embodiment of the present invention provides a method for detecting early wheat stripe rust, which can be implemented by an electronic device, and the electronic device can be a terminal or a server. Figure 1 The flowchart of the method is shown as follows, Figure 2 The overall block diagram of the method is shown as follows, and the processing flow may include the following steps:

[0071] S1. Obtain and preprocess multi-modal image data and corresponding multi-modal time data of early and late wheat, comprehensively use the early data to make detection box labels, and divide them into a training set and a validation set;

[0072] Optionally, S1 specifically includes:

[0073] Use a drone to carry red light, near-infrared light, visible light, and polarized light cameras to photograph the entire wheat field, and obtain red light image data D of early wheat stripe rust for multiple consecutive days (such as 5 days) 1 , near-infrared light image data D 2 , visible light image data D 3 , polarized light image data D 4 and early time data T 1 . When the stripe rust enters the late disease stage with obvious characteristics, use the drone to photograph the entire wheat field again, and obtain red light image data W of late wheat stripe rust for multiple consecutive days (such as 5 days) 1 , near-infrared light image data W 2 , visible light image data W 3 , polarized light image data W 4 and late time data T 2 ;

[0074] Perform spatial position registration on the early stripe rust image data D 1 , D 2 , D 3 , D 4 and the late stripe rust image data W 1 , W 2 , W 3 , W 4 respectively. Through geometric correction, ensure that the resolutions, projections, and coordinate systems of all modal images are consistent, eliminate the errors caused by the shooting angle, and provide consistent spatial information for subsequent data annotation and model training;

[0075] By observing the manifestations of early stripe rust symptoms at different times in different modalities (for example, red light and near-infrared light modalities can reveal the health status of plants, visible light can characterize the surface texture changes of plants, and polarized light can enhance the contrast of fine lesions), combined with visible light, polarized light, near-infrared light, and red light image data, detection box labels are made for the early multi-modal image data and divided into a training set and a validation set (for example, it can be divided according to 9:1).

[0076] S2. Input the early multi-modal image data, late multi-modal image data, and corresponding multi-modal time data in the training set into the time feature fusion module, fully extract and fuse the multi-modal data features at different time periods, obtain a richer and more comprehensive feature representation of stripe rust than single-modal data, and obtain multi-modal data embedded with the time dimension;

[0077] Optionally, S2 specifically includes:

[0078] Respectively input the early data (D 1 , D 2 , D 3 , D 4 , T 1 ) and the late data (W 1 , W 2 , W 3 , W 4 , T 2 ) into the time feature fusion module to obtain the early image data (D′ 1 , D′ 2 , D′ 3 , D′ 4 ) and the late image data (W′ 1 , W′ 2 , W′ 3 , W′ 4 ) embedded with the time dimension. Among them, as Figure 3 shown, the time feature fusion module passes the multi-modal image data through a 3×3 convolution repeated 3 times and a normalization layer, ReLU activation function, 3×3 convolution, and normalization layer connected by a 1×1 convolution residual, performs cross-attention operation on the operation result and the multi-spectral time data after linear mapping, and the operation result passes through a normalization layer with residual connection repeated 2 times, Dropout and normalization layer, feed-forward network with residual connection, normalization layer, and feed-forward network, and outputs the multi-modal data embedded with the time dimension.

[0079] S3, input the multimodal data embedded in the time dimension and pre-train the spectral reconstruction and time series contrast learning model, dig out the general and common feature patterns of wheat stripe rust at different development stages and in different spectral bands, the spectral reconstruction and time series contrast learning model includes two parallel branches, one branch randomly covers part of the image information and performs spectral reconstruction training, by letting the feature reconstructor in the model learn to restore the covered information, forcing the feature extractor in the model to learn the general disease space features; the other branch constructs a comparison sample pair after data enhancement operation, performs time series contrast training, and uses contrast loss to force the feature extractor in the model to learn to map data in the same time period to a similar feature space, and data in different time periods to a farther feature space, and uses time information to force the model to distinguish the stripe rust features at different time points, prompting the feature extractor to more keenly capture the early features of stripe rust;

[0080] Alternatively, if Figure 4 As shown, the spectral reconstruction training in S3 specifically includes:

[0081] The input early and late multimodal data embedded in the time dimension are spliced ​​according to the channel dimension, and the spliced ​​image is cut into image blocks according to the size of 16×16 pixels by spectral channel;

[0082] Then, 50% of the image blocks are randomly masked and position encoding information is added, and then input into the feature extractor to extract the spectral features represented by the unmasked image blocks;

[0083] Then the extracted spectral features are input into the feature reconstructor, and the masked image blocks are reconstructed using the spectral features extracted by the feature extractor. The mean square error (MSE) is used as the spectral reconstruction loss to calculate the difference between the pixels of the image block reconstructed by the feature reconstructor and the pixels of the input image block. By using the spectral reconstruction loss to supervise the model learning, the model's perception of the spectral information of late stripe rust is enhanced, and a universal feature extractor that can handle multiple modal spectral inputs is obtained.

[0084] The temporal contrast training in S3 specifically includes:

[0085] Perform two random data augmentation operations on the early and late data respectively to obtain the augmented data 1(D′ 1.1 , D′ 2.1 , D′ 3.1 , D′ 4.1 , W′ 1.1 , W′ 2.1 , W′ 3.1 , W′ 4.1 ) and enhanced data 2 (D′ 1.2 , D′ 2.2 , D′ 3.2 , D′4.2 , W' 1.2 , W' 2.2 , W' 3.2 , W' 4.2 );

[0086] Then, the 16 enhanced data in two parts are combined in pairs to construct 120 non-repetitive comparison sample pairs. After the comparison sample pairs are extracted with features by the feature extractor and then processed by the linear layer, activation function and normalization layer, 120 comparison feature pairs L 1 , l 2 , l 3 ,......, L 120 , Calculate the similarity between each comparison sample pair through cosine, and then calculate the temporal contrast loss through the normalized cross-entropy loss function N-Xent. The calculation formula is as follows:

[0087]

[0088]

[0089] Through temporal contrast learning training, the model can understand the variation law of the spectral characteristics of stripe rust over time;

[0090] A joint loss function composed of mean squared error and normalized cross-entropy loss function is used to backpropagate and optimize the parameters of the feature extractor and feature reconstructor, so that the model can capture spatio-temporal joint features, and at the same time consider the disease manifestations in space and the evolution process in time.

[0091] Optionally, as Figure 5 shown, the feature extractor includes 12 layers, and each layer consists of a linear layer, a normalization layer with a 3×3 convolutional residual connection and a self-attention layer, plus a normalization layer with a 3×3 convolutional residual connection and a multi-layer shared perceptron;

[0092] As Figure 6 shown, on the one hand, for the feature reconstructor, the input image is processed by the difference between max pooling and average pooling to screen out the parts with high feature values in the input image. After preliminary feature extraction by a 3×3 convolution, it is input into the activation function to obtain the weight matrix of the pixels in different regions of the image. The input image is multiplied by the weight matrix after passing through a 3×3 convolution to focus on the local regions with important information in the image and learn local features;

[0093] On the other hand, the input image dynamically adjusts the attention degree of each input position to other positions through layer normalization, self-attention mechanism and 3×3 convolution, integrates global information to form a more accurate representation, and learns global features;

[0094] After concatenating the learned local features and global features, input them into a multi-layer perceptron to output the reconstructed feature image.

[0095] S4. Transfer the weights of the pre-trained feature extractor to the multi-modal encoder of the multi-modal multi-scale object detection model (the multi-modal encoder loads the pre-trained weights trained with early stripe rust data using embedded time information as prior knowledge and freezes the parameters). The multi-modal encoder automatically adjusts the weights of different modal features in the feature space, performs more refined multi-modal feature extraction and fusion, extracts interactive fusion features of different scales, and inputs them into the multi-scale decoder of the multi-modal multi-scale object detection model. The multi-scale decoder introduces high-level features learned at low resolution into high-resolution features through skip connections, comprehensively utilizes global context information and local detail information, and at the same time makes full use of features of different scales through the cross-attention mechanism to output the detection results of stripe rust;

[0096] Optionally, as Figure 7 shown, the multi-modal encoder consists of three repeatedly stacked multi-modal coding networks. Each multi-modal coding network consists of four parallel sub-feature extractors with the same weights and a multi-modal fusion network. The sub-feature extractors E 1 , E 2 and E 3 load the weights of the 1st - 4th layers, 5th - 8th layers, and 9th - 12th layers of the pre-trained feature extractor respectively, for extracting multi-modal features and inputting them into the multi-modal fusion network;

[0097] As Figure 8 shown, the multi-modal fusion network consists of interactive modal fusion modules. The interactive modal fusion module, as Figure 9 shown, the input feature Figure 1 and feature Figure 2 respectively obtain query vector Q 1 , key vector K 1 , value vector V 1 and query vector Q 2 , key vector K 2 , value vector V 2 through linear mapping. The query vector Q Figure 1 of feature 1 multiplies with the transpose of the key vector K Figure 2 of feature 2 to obtain a weight matrix. After processing the weight matrix through sigmoid and multiplying it with the value vector V Figure 2 of feature 2 , a weighted matrix of the attention degree of different parts of feature Figure 1 to feature Figure 2 is obtained; The query vector Q Figure 2The query vector Q 1 is multiplied by the transpose of the key vector K Figure 1 of the feature, and after sigmoid processing, it is multiplied by the value vector V 1 of the feature Figure 1 to obtain the feature 1 A weighted matrix for the attention degrees of different parts of the feature Figure 2 is obtained; the two weighted matrices and the feature Figure 1 are concatenated by channel and then subjected to a 1×1 convolution operation to obtain a fused feature; Figure 1 and the feature Figure 2 In multi-modal fusion network 1 and multi-modal fusion network 2, as

[0098] shown, first, the red light and the near-infrared light are fused through an interactive modality fusion module, the obtained fusion result is fused with the visible light in the interactive modality fusion module, and then the new fusion result is fused with the polarized light through the interactive modality fusion module again. One side of the fusion result is output as the interactive fusion feature M, and the other side is respectively subjected to feature concatenation in the channel dimension with the original red light, near-infrared light, visible light, and polarized light feature maps, and then input into a deep multi-modal encoding network after 2-fold downsampling as input features. The multi-modal fusion network 3 only outputs the interactive fusion feature M and no longer outputs the input of the deep network. Three different spatial scale interactive fusion features M Figure 8 are obtained through three groups of multi-modal encoding networks 1 , M 2 , M 3 , as Figure 7 shown.

[0099] Optionally, as Figure 10 shown, the input of the multi-scale decoder is the interactive fusion feature M 1 , M 2 , M 3 ;

[0100] M 3 is subjected to a 3×3 convolution to obtain the interactive fusion feature M' 3 ;

[0101] M 3 is subjected to 2-fold upsampling and then concatenated by channel with M 2 and a 1×1 convolution operation is performed to obtain M 2,3 , and then subjected to a 3×3 convolution to obtain the interactive fusion feature M' 2 ;

[0102] M 2 is subjected to 2-fold upsampling and then concatenated by channel with M 1 and a 1×1 convolution operation is performed to obtain M 1,2 , and then concatenated by channel with M 2,3The results after 2x upsampling are concatenated by channel and subjected to a 1x1 convolution operation to obtain M 1,2,3 and then subjected to a 3x3 convolution operation to obtain the interactive fusion feature M' 1 ;

[0103] The interactive fusion features M' at different scales 1 , M' 2 , M' 3 After being flattened, are input into a linear mapping layer for mapping to form feature Tokens. The feature Tokens obtained from feature maps of different scales are concatenated according to the dimension of the number of Tokens. The result is used as the value vector, and the concatenated feature Tokens are used as the key vector after adding position encoding. At the same time, N target query vectors are randomly initialized according to a Gaussian distribution, where N is the upper limit of the prediction instance targets that the model can give;

[0104] After performing cross-attention operations on the key, value, and target query vectors to obtain a feature attention matrix, the feature attention matrix is layer-normalized, then input into a self-attention layer with residual connections and layer-normalized, and then the operation result is input into a feed-forward neural network with residual connections for layer-normalization again, thus completing one query process. After repeating the query process 6 times, the predicted query feature is obtained. The predicted query feature is input into two parallel feed-forward neural networks, and is respectively mapped to the predicted values of the detection box coordinates and the confidence level through linear layers.

[0105] S5. Use the labeled data in the early stripe rust data to fine-tune the multi-scale decoder parameters of the multi-modal multi-scale object detection model to obtain a trained multi-modal multi-scale object detection model;

[0106] Optionally, the specific steps of using the labeled data in the early stripe rust data to fine-tune the multi-scale decoder parameters of the multi-modal multi-scale object detection model in S5 include:

[0107] The parameters of the decoder of the multi-modal multi-scale object detection model are randomly initialized with weights based on a Gaussian distribution;

[0108] Input the labeled early stripe rust data into the multi-modal multi-scale object detection model, calculate the intersection over union between each detection box label and the detection box prediction value, screen out the detection box prediction value with the largest intersection over union, and use the MSE loss function to calculate the loss between the confidence level prediction value and the value 1, and the loss between the detection box prediction value with the largest intersection over union and the detection box label. The sum of the two is used as the total loss for backpropagation to optimize the weights of the multi-scale decoder.

[0109] S6. Use the trained multi-modal multi-scale object detection model to detect early stripe rust in the wheat to be detected.

[0110] Obtain the multi-modal image data of the wheat to be detected on the current day, and input it into the trained multi-modal multi-scale object detection model to detect and output the predicted values of the detection box coordinates and the predicted values of the confidence levels.

[0111] As Figure 11 shown, an embodiment of the present invention also provides a wheat early stripe rust detection system, and the system includes:

[0112] An acquisition and preprocessing module 1110, configured to acquire and preprocess the multi-modal image data and corresponding multi-modal time data of wheat in the early and late stages, comprehensively use the early data to make detection box labels, and divide them into a training set and a validation set;

[0113] A time feature fusion module 1120, configured to input the early multi-modal image data, late multi-modal image data and corresponding multi-modal time data in the training set into the time feature fusion module, fully extract and fuse the multi-modal data features of different time periods, obtain a richer and more comprehensive feature representation of stripe rust than single-modal data, and obtain multi-modal data embedded with time dimension;

[0114] A pre-training module 1130, configured to input the multi-modal data embedded with time dimension and pre-train a spectral reconstruction and temporal contrast learning model, and mine the general and common feature patterns shown by wheat stripe rust at different development stages and in different spectral bands. The spectral reconstruction and temporal contrast learning model includes two parallel branches. After randomly covering part of the image information in one branch, spectral reconstruction training is carried out. By allowing the feature reconstructor in the model to learn to restore the covered information, the feature extractor in the model is forced to learn general disease space features; the other branch constructs comparison sample pairs after data augmentation operations and performs temporal contrast training. By using the contrast loss, the feature extractor in the model is forced to learn to map the data in the same time period to a similar feature space, and the data in different time periods are mapped to a farther feature space. The time information is used to force the model to distinguish the stripe rust features at different time points, and the feature extractor is prompted to more sensitively capture the early features of stripe rust;

[0115] An object detection module 1140, configured to transfer the weights of the pre-trained feature extractor to the multi-modal encoder of the multi-modal multi-scale object detection model. The multi-modal encoder automatically adjusts the weights of different modal features in the feature space, performs more refined multi-modal feature extraction and fusion, extracts interactive fusion features of different scales, and inputs them into the multi-scale decoder of the multi-modal multi-scale object detection model. The multi-scale decoder introduces the high-level features learned at low resolution into the high-resolution features through skip connections, comprehensively uses the global context information and local detail information, and at the same time makes full use of the features of different scales through the cross-attention mechanism to output the detection results of stripe rust;

[0116] A fine-tuning module 1150, configured to fine-tune the multi-scale decoder parameters of the multi-modal multi-scale object detection model with the labeled data in the early stripe rust data, so as to obtain a trained multi-modal multi-scale object detection model;

[0117] A detection module 1160, configured to use the trained multi-modal multi-scale object detection model to perform early stripe rust detection on the wheat to be detected.

[0118] A wheat early stripe rust detection system provided by an embodiment of the present invention has a functional structure corresponding to a wheat early stripe rust detection method provided by an embodiment of the present invention, which will not be elaborated herein.

[0119] Figure 12 FIG. 11 is a schematic structural diagram of an electronic device 1200 provided by an embodiment of the present invention. The electronic device 1200 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPUs) 1201 and one or more memories 302. Among them, at least one instruction is stored in the memory 302, and the at least one instruction is loaded and executed by the processor 1201 to implement the steps of the above-mentioned wheat early stripe rust detection method.

[0120] In an exemplary embodiment, a computer-readable storage medium is further provided, such as a memory including instructions, and the above instructions can be executed by a processor in a terminal to complete the above-mentioned wheat early stripe rust detection method. For example, the computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0121] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above embodiments can be completed by hardware, or can be completed by instructing relevant hardware through a program. The program can be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disc, etc.

[0122] The above are only the preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for detecting early wheat stripe rust, characterized in that: The method comprises: S1, obtaining and preprocessing early and late multimodal image data of wheat and corresponding multimodal time data, comprehensively utilizing early data to produce detection frame labels, and dividing them into training sets and validation sets; S2, inputting the early multimodal image data, late multimodal image data and corresponding multimodal time data in the training set into the time feature fusion module, fully extracting and fusing the multimodal data features of different time periods, obtaining a richer and more comprehensive feature representation of stripe rust than single modal data, and obtaining multimodal data embedded in the time dimension; S3, input the multimodal data embedded in the time dimension and pre-train the spectral reconstruction and time series contrast learning model, dig out the general and common feature patterns of wheat stripe rust at different development stages and in different spectral bands, the spectral reconstruction and time series contrast learning model includes two parallel branches, one branch randomly covers part of the image information and performs spectral reconstruction training, by letting the feature reconstructor in the model learn to restore the covered information, forcing the feature extractor in the model to learn the general disease space features; the other branch constructs a comparison sample pair after data enhancement operation, performs time series contrast training, and uses contrast loss to force the feature extractor in the model to learn to map data in the same time period to a similar feature space, and data in different time periods to a farther feature space, and uses time information to force the model to distinguish the stripe rust features at different time points, prompting the feature extractor to more keenly capture the early features of stripe rust; S4, transferring the weights of the pre-trained feature extractor to the multimodal encoder of the multimodal multi-scale target detection model, the multimodal encoder automatically adjusts the weights of different modal features in the feature space, performs more sophisticated multimodal feature extraction and fusion, extracts interactive fusion features of different scales, and inputs them into the multiscale decoder of the multimodal multi-scale target detection model. The multiscale decoder introduces high-level features learned at low resolution into high-resolution features through skip connections, comprehensively utilizes global context information and local detail information, and fully utilizes features of different scales through a cross-attention mechanism to output the detection results of stripe rust; S5. Fine-tune the multi-scale decoder parameters of the multi-modal multi-scale target detection model using the labeled data in the early stripe rust data to obtain a trained multi-modal multi-scale target detection model; S6. Use the trained multi-modal and multi-scale target detection model to perform early stripe rust detection on the wheat to be detected.

2. The method according to claim 1, characterized in that The S1 specifically includes: The whole wheat field is photographed by a drone equipped with red light, near infrared light, visible light and polarized light cameras to obtain red light image data D1, near infrared light image data D2, visible light image data D3, polarized light image data D4 and early time data T1 of wheat stripe rust for multiple consecutive days in the early stage. When the stripe rust enters the late stage with obvious characteristics, the whole wheat field is photographed again by a drone to obtain red light image data W1, near infrared light image data W2, visible light image data W3, polarized light image data W4 and late time data T2 of wheat stripe rust for multiple consecutive days in the late stage. The early stripe rust image data D1, D2, D3, D4 and the late stripe rust image data W1, W2, W3, W4 are spatially registered respectively, and the resolution, projection and coordinate system of all modal images are ensured to be consistent through geometric correction, so as to eliminate the error caused by the shooting angle and provide consistent spatial information for subsequent data annotation and model training; By observing the manifestations of early stripe rust symptoms in different modalities at different times, and combining visible light, polarized light, near-infrared light, and red light image data, detection frame labels are created for the early multimodal image data and divided into training and validation sets.

3. The method according to claim 2, characterized in that The S2 specifically includes: The early data (D1, D2, D3, D4, T1) and the late data (W1, W2, W3, W4, T2) are respectively input into the time feature fusion module to obtain the early image data (d'1, d'2, d'3, D'4) and the late image data (w'1, w'2, w'3, w'4) embedded in the time dimension, wherein the time feature fusion module subjects the multimodal image data to 3×3 convolutions repeatedly stacked 3 times and a normalization layer connected by a 1×1 convolution residual, a ReLU activation function, a 3×3 convolution and a normalization layer, and performs a cross-attention operation on the operation result and the multispectral time data after linear mapping, and the operation result passes through a normalization layer with a residual connection repeatedly stacked 2 times, a Dropout and normalization layer, a feedforward network with a residual connection, a normalization layer and a feedforward network, and outputs the multimodal data embedded in the time dimension.

4. The method according to claim 1, characterized in that: The spectral reconstruction training in S3 specifically includes: The input early and late multimodal data embedded in the time dimension are spliced ​​according to the channel dimension, and the spliced ​​image is cut into image blocks according to the size of 16×16 pixels by spectral channel; Then, 50% of the image blocks are randomly masked and position encoding information is added, and then input into the feature extractor to extract the spectral features represented by the unmasked image blocks; Then the extracted spectral features are input into the feature reconstructor, and the masked image blocks are reconstructed using the spectral features extracted by the feature extractor. The mean square error (MSE) is used as the spectral reconstruction loss to calculate the difference between the pixels of the image block reconstructed by the feature reconstructor and the pixels of the input image block. By using the spectral reconstruction loss to supervise the model learning, the model's perception of the spectral information of late stripe rust is enhanced, and a universal feature extractor that can handle multiple modal spectral inputs is obtained. The temporal contrast training in S3 specifically includes: Perform two random data augmentation operations on the early and late data respectively to obtain the enhanced data 1 (D' 1.1 , d' 2.1 , d' 3.1 , d' 4.1 , W' 1.1 , W' 2.1 , W' 3.1 , W' 4.1 ) and enhanced data 2 (D' 1.2 , d' 2.2 , D' 3.2 , D' 4.2 , W' 1.2 , W' 2.2 , W' 3.2 , W' 4.2 ); Then, the two parts of 16 enhanced data are combined in pairs to construct 120 non-repetitive comparison sample pairs. After the comparison sample pairs are extracted by the feature extractor, they are processed by the linear layer, activation function and normalization layer to obtain 120 comparison feature pairs L1, L2, L3, ..., L 120 , the similarity between each comparison sample pair is calculated by cosine, and then the temporal contrast loss is calculated by the normalized cross entropy loss function N-Xent. The calculation formula is as follows: Through time series contrast learning training, the model can understand the temporal changes of the spectral characteristics of stripe rust; The joint loss function is composed of the mean square error and normalized cross entropy loss functions. Back propagation optimizes the parameters of the feature extractor and feature reconstructor, so that the model can capture the joint spatiotemporal features while considering the spatial manifestations of the disease and the temporal evolution process.

5. The method according to claim 1, characterized in that The feature extractor includes 12 layers, each of which consists of a linear layer, a 3×3 convolutional residual connected normalization layer and a self-attention layer, plus a 3×3 convolutional residual connected normalization layer and a multi-layer shared perceptron; The feature reconstructor, on the one hand, processes the difference of maximum pooling and flat pooling on the input image, screens out the part with high eigenvalue in the input image, and then extracts the features through 3×3 convolution, and then inputs the activation function to obtain the weight matrix of pixels in different regions of the image. The input image is multiplied by the weight matrix after 3×3 convolution, focuses on the local region with important information in the image, and learns the local features; On the other hand, the input image is layer-normalized, self-attention mechanism and 3×3 convolution dynamically adjust the attention of each input position to other positions, integrate global information to form a more accurate representation, and learn global features; After the learned local features and global features are concatenated, they are input into a multi-layer perceptron to output the reconstructed feature image.

6. The method according to claim 1, characterized in that The multimodal encoder is composed of three repeatedly stacked multimodal encoding networks, each of which is composed of four parallel sub-feature extractors with the same weight and a multimodal fusion network. The sub-feature extractors E1, E2 and E3 of the three multimodal encoding networks are respectively loaded with the weights of the 1st to 4th layers, 5th to 8th layers and 9th to 12th layers of the pre-trained feature extractors, so as to extract multimodal features and input them into the multimodal fusion network. The multimodal fusion network is composed of an interactive modal fusion module. In the interactive modal fusion module, the input feature map 1 and feature map 2 are linearly mapped to obtain query vector Q1, key vector K1, value vector V1 and query vector Q2, key vector K2, value vector V2 respectively. The query vector Q1 of feature map 1 is multiplied by the transpose of the key vector K2 of feature map 2 to obtain a weight matrix. The weight matrix is ​​multiplied by the value vector V2 of feature map 2 after sigmoid processing to obtain a weighted matrix of the degree of attention of feature map 1 to different parts of feature map 2; the query vector Q1 of feature map 2 is multiplied by the transpose of the key vector K1 of feature map 1, and is multiplied by the value vector V1 of feature map 1 after sigmoid processing to obtain a weighted matrix of the degree of attention of feature map 2 to different parts of feature map 1; the two weighted matrices are spliced ​​with feature map 1 and feature map 2 according to channels and then a 1×1 convolution operation is performed to obtain a fusion feature; In multimodal fusion network 1 and multimodal fusion network 2, red light and near-infrared light are first fused through the interactive modal fusion module, the fusion result is fused with visible light in the interactive modal fusion module, and then the new fusion result is fused with polarized light through the interactive modal fusion module. The fusion result is output as the interactive fusion feature M on one side, and on the other side, it is concatenated with the original red light, near-infrared light, visible light and polarized light feature maps in the channel dimension, and then input into the deep multimodal encoding network as the input feature after 2 times downsampling. Multimodal fusion network 3 only outputs the interactive fusion feature M, and no longer outputs the input of the deep network. After three groups of multimodal encoding networks, three interactive fusion features M1, M2, and M3 of different spatial scales are obtained.

7. The method according to claim 6, characterized in that The multi-scale decoder, input is the interactive fusion features M1, M2, M3; M3 undergoes 3×3 convolution to obtain the interactive fusion feature M'3; After 2 times upsampling, M3 is concatenated with M2 by channel and performs 1×1 convolution operation to obtain M 2,3 , and then through 3×3 convolution to obtain the interactive fusion feature M'2; After 2 times upsampling, M2 is concatenated with M1 by channel and performs 1×1 convolution operation to obtain M 1,2 , then with M 2,3 After 2 times upsampling, the results are concatenated by channel and subjected to 1×1 convolution operation to obtain M 1,2,3 , and then through 3×3 convolution operation to obtain the interactive fusion feature M'1; After the interactive fusion features M'1, M'2, and M'3 of different scales are flattened, they are input into the linear mapping layer for mapping and mapped into feature tokens. The feature tokens obtained from the feature maps of different scales are concatenated according to the dimension of the number of tokens, and the result is used as the value vector. The concatenated feature tokens are added to the position code as the key vector, and N target query vectors are randomly initialized according to the Gaussian distribution. N is the upper limit of the predicted instance target that the model can give. A feature attention matrix is ​​obtained after cross-attention operation on the key, value and target query vectors. After layer normalization, the feature attention matrix is ​​input into the self-attention layer with residual connection and layer normalization is performed. Then the operation result is input into the feedforward neural network with residual connection and layer normalization is performed again. Thus, a query process is completed. The query process is repeated 6 times to obtain the predicted query features. The predicted query features are input into two parallel feedforward neural networks and mapped into detection box coordinate prediction values ​​and confidence prediction values ​​through linear layers respectively.

8. A wheat early stripe rust detection system, characterized in that: The system comprises: A preprocessing module is obtained to obtain and preprocess the multimodal image data of early and late wheat and the corresponding multimodal time data, comprehensively utilize the early data to make detection frame labels, and divide them into training sets and validation sets; The temporal feature fusion module is used to input the early multimodal image data, late multimodal image data and corresponding multimodal time data in the training set into the temporal feature fusion module, fully extract and fuse the multimodal data features of different time periods, obtain a richer and more comprehensive feature representation of stripe rust than single modal data, and obtain multimodal data embedded in the time dimension; A pre-training module is used to input the multimodal data embedded in the time dimension and pre-train a spectral reconstruction and time series contrast learning model to mine general and common feature patterns of wheat stripe rust at different development stages and in different spectral bands. The spectral reconstruction and time series contrast learning model includes two parallel branches. One branch randomly masks part of the image information and then performs spectral reconstruction training. By allowing the feature reconstructor in the model to learn to restore the masked information, the feature extractor in the model is forced to learn the general disease space features. The other branch constructs a comparison sample pair after data enhancement operation and performs time series contrast training. By using contrast loss, the feature extractor in the model is forced to learn to map data in the same time period to a similar feature space, and data in different time periods to a farther feature space. The time information is used to force the model to distinguish the stripe rust features at different time points, so that the feature extractor can more keenly capture the early features of the stripe rust. The target detection module is used to transfer the weights of the pre-trained feature extractor to the multimodal encoder of the multimodal multi-scale target detection model. The multimodal encoder automatically adjusts the weights of different modal features in the feature space, performs more sophisticated multimodal feature extraction and fusion, extracts interactive fusion features of different scales, and inputs them into the multiscale decoder of the multimodal multi-scale target detection model. The multiscale decoder introduces high-level features learned at low resolution into high-resolution features through skip connections, comprehensively utilizes global context information and local detail information, and fully utilizes features of different scales through a cross-attention mechanism to output the detection results of stripe rust; A fine-tuning module, used to fine-tune the multi-scale decoder parameters of the multi-modal multi-scale target detection model using labeled data in the early stripe rust data to obtain a trained multi-modal multi-scale target detection model; The detection module is used to use the trained multi-modal and multi-scale target detection model to perform early stripe rust detection on the wheat to be detected.

9. An electronic device, comprising a processor and a memory, wherein at least one instruction is stored in the memory, wherein: The at least one instruction is loaded and executed by the processor to implement the method for detecting early wheat stripe rust according to any one of claims 1 to 7.

10. A computer-readable storage medium, wherein at least one instruction is stored in the storage medium, characterized in that: The at least one instruction is loaded and executed by the processor to implement the method for detecting early wheat stripe rust according to any one of claims 1 to 7.

Images