Self-supervised mulberry disease and insect pest identification method and system fused with multi-mode YOLO algorithm

Abstract

The invention provides a self-supervised mulberry disease and insect pest recognition method and system fused with a multi-mode YOLO algorithm, and the method comprises the steps: providing a Multi-Modan-YOLO algorithm fused with a multi-mode information processing technology, namely, the multi-mode YOLO algorithm, and carrying out the self-supervised mulberry disease and insect pest recognition method fused with the multi-mode YOLO algorithm; and audio information and thermal imaging information are combined to further improve the accuracy of the identification result of the pest and disease damage types. A self-supervised learning framework driven by a multi-modal large model is introduced, when the identification deviation risk exceeds a preset threshold value, the multi-modal large model is called to generate a category label, and the model is endowed with the ability of dynamically generating a new label for a new type of pest and disease damage. Therefore, misrecognition caused by non-uniform distribution of training data or missing of a certain category of data is effectively prevented; according to the method, targets such as disease spots and insect pests on mulberry leaves are accurately identified, so that the efficiency and accuracy of mulberry disease and insect pest prevention and control are improved, the influence on silkworm production is reduced, and yield increase is promoted.

Description

Technical Field

[0001] The present invention relates to the field of smart agricultural technology, and in particular to a self-supervised mulberry tree disease and insect pest identification method integrating a multimodal YOLO algorithm. Background Art

[0002] At present, it is not difficult to identify whether there are pests and diseases in sericulture production using existing technologies, but identifying the specific pests and diseases is still a certain challenge. Traditional methods for detecting mulberry pests and diseases usually rely on manual observation and empirical judgment. This method is not only time-consuming and labor-intensive, but also easily affected by subjective factors, resulting in low recognition accuracy. With the development of machine learning and computer vision technology, the use of image recognition algorithms for mulberry pest and disease detection has gradually become an effective solution. However, single-modality image data, such as RGB images, are affected by factors such as lighting conditions and environmental complexity, and there is still a problem of low accuracy in identifying the types of pests and diseases.

[0003] Among the pests and diseases suffered by mulberry trees, some insects may emit audio information in specific patterns. At the same time, thermal imaging data can better detect other organisms on mulberry trees or mulberry leaves, especially those with colors similar to the background. On the other hand, some pathogenic microorganisms may change the appearance or shape of mulberry leaves, and this information representing the shape is easier to extract in thermal imaging data. Therefore, multimodal amplification based on RGB image data can effectively improve the recognition accuracy of pest and disease categories. Through multimodal data fusion and self-supervised learning technology, the adaptability of the model in the absence of labeled data can be further improved, and new pest and disease category labels can be dynamically generated, thereby improving the robustness and generalization ability of the pest and disease detection system.

[0004] Most existing pest and disease detection models rely on labeled training data, which has great limitations under the realistic conditions where the labeling cost is high and it is difficult to cover all types of pests and diseases. Especially when faced with newly emerging or rare pests and diseases, traditional supervised learning models often cannot provide accurate detection results. In addition, due to the wide variety of pests and diseases and their complex and diverse manifestations, single-modality image data, such as RGB, cannot fully capture the characteristics of different pests and diseases. This requires recognition methods that can combine multi-modal data, such as thermal imaging, audio signals, etc., to improve the detection accuracy and adaptability of the system.

[0005] In order to overcome these challenges, self-supervised learning has become an effective way to solve the problem of insufficient labeled data in recent years. Self-supervised learning driven by multimodal large models can perform feature learning on unlabeled data by constructing pseudo-labels or utilizing the intrinsic structure of the data, thereby enhancing the model's ability to identify new types of pests and diseases. In addition, the fusion of multimodal YOLO algorithms contributes to the complementarity of data in different modalities, further improving the robustness and accuracy of pest and disease identification. Through this innovative approach, the system can not only effectively address the data labeling bottleneck faced by traditional supervised learning models, but also achieve accurate detection and classification in complex and diverse pest and disease environments.

[0006] In summary, it is necessary to provide a self-supervised mulberry pest and disease identification method integrating the multimodal YOLO algorithm to solve the above technical problems. Summary of the invention

[0007] In order to solve the above technical problems, the present invention provides a self-supervised mulberry tree disease and insect pest identification method and system integrating a multimodal YOLO algorithm.

[0008] The present invention provides a self-supervised mulberry tree pest and disease identification method integrating a multimodal YOLO algorithm, the method comprising the following steps:

[0009] S1: Information collection and model training, using sensors to obtain multimodal data of mulberry leaves, including RGB color images, thermal imaging temperature distribution data, and audio data. Using the constructed RGB color image dataset and its label information, the image feature calculation module in the constructed multimodal YOLO algorithm model is trained. The multimodal YOLO algorithm model consists of an image feature calculation module, a multimodal feature extraction module, a cross-modal feature fusion module, and a self-supervised label generation module;

[0010] S2: extracting image features based on the YOLO algorithm, calling the image feature calculation module in the trained multimodal YOLO algorithm model, recognizing the collected RGB color image, and outputting the obtained image modality decision vector. The image feature calculation module in the multimodal YOLO algorithm model is a YOLOv9 model suitable for image classification tasks;

[0011] S3: Multimodal feature extraction: input the thermal imaging temperature distribution data and audio data into the multimodal feature extraction module in the multimodal YOLO algorithm model. The multimodal feature extraction module in the multimodal YOLO algorithm model is a graph convolutional neural network model suitable for feature extraction tasks. Feature extraction is performed through the multimodal feature extraction module in the multimodal YOLO algorithm model to generate a unified multimodal feature vector;

[0012] S4: Cross-modal feature fusion, feature fusion of the image modality decision vector output by the image feature calculation module in the multimodal YOLO algorithm model and the multimodal feature vector output by the multimodal feature extraction module, and outputting a one-dimensional decision vector, wherein the cross-modal feature fusion module in the multimodal YOLO algorithm model is a Boltzmann machine neural network model, and the one-dimensional decision vector is a 1×N-dimensional array, where N represents the number of state categories, and the sum of all values ​​in the array is 1. The first value in the one-dimensional decision vector represents the probability that the current data sample belongs to the normal category, the second value represents the probability that the current data sample belongs to the mulberry shriveling disease category, and the third value represents the probability that the current data sample belongs to the mulberry blight disease category, and each number therein represents the probability of a state category;

[0013] S5: Identification bias risk calculation, calculating the identification bias risk of the one-dimensional decision vector output by the multimodal YOLO algorithm. When it exceeds the preset threshold input by the user, go to step S6; otherwise, go to step S7;

[0014] S6: self-supervised label generation, calling the self-supervised label generation module in the multimodal YOLO algorithm model to generate category labels, giving the model the ability to dynamically generate new labels for new types of pests and diseases, thereby effectively preventing misidentification caused by uneven distribution of training data or missing data of a certain category. The self-supervised label generation module in the multimodal YOLO algorithm model is a large multimodal model used to process input image information to generate a text label for the current sample;

[0015] S7: Sample category decision: if the self-supervised label generation module in the multimodal YOLO algorithm model has a new label generated for the current sample, the new label is extracted; otherwise, the maximum value of the category probability is extracted from the one-dimensional decision vector, and the sample label corresponding to the value is extracted. The extracted sample label is output as the decision result.

[0016] In order to construct a multimodal YOLO algorithm model for intelligent identification of mulberry pests and diseases, in step S1, the constructed multimodal YOLO algorithm model includes an image feature calculation module, a multimodal feature extraction module, a cross-modal feature fusion module, and a self-supervised label generation module. Among them, the image feature calculation module in the multimodal YOLO algorithm model is a YOLOv9 model suitable for image classification tasks, used to extract feature information from sample image data; the multimodal feature extraction module is a graph convolutional neural network model suitable for feature extraction tasks, used to extract key features from multimodal information; the cross-modal feature fusion module is a Boltzmann machine neural network model suitable for prediction tasks, used to fuse the features of multimodal data such as RGB color images, thermal imaging temperature distribution data, and audio data; the self-supervised label generation module is a large multimodal model suitable for text generation tasks, used to process input image information to generate a text label for the current sample;

[0017] In order to extract feature data of multimodal information for identifying mulberry tree pests and diseases, in step S3, multimodal feature extraction is performed according to the following steps:

[0018] Step 3-1: Represent the collected thermal imaging temperature distribution data as a two-dimensional matrix Where T represents the thermal imaging temperature distribution matrix, m represents the width of the thermal imaging data, and n represents the height of the thermal imaging data. represents a matrix consisting of real numbers, represents a matrix of width m and height n composed of real numbers, each element of which represents the temperature value of a certain position of a mulberry leaf;

[0019] Step 3-2: Input the thermal imaging temperature distribution matrix T into the multimodal feature extraction module and perform feature extraction through the graph convolutional neural network model. This feature extraction can be described as:

[0020]

[0021] in, is the node feature matrix of the lth layer, is the node feature matrix of the lth layer, l is the current number of layers of the node feature matrix, represents the initial state of the node feature matrix, T represents the thermal imaging temperature distribution matrix. A is the adjacency matrix, which represents the spatial structure of the temperature data, D is the degree matrix, and i and j represent the row vector pointer and column vector pointer respectively. is the weight matrix of the lth layer, σ(·) is the ReLU activation function;

[0022] Step 3-3: After the multi-layer convolution operation of the convolutional neural network, the final temperature data features extracted meet the following requirements:

[0023]

[0024] where d T Represents the characteristic dimension of the extracted temperature data, H T represents the node feature matrix, and T represents the thermal imaging temperature distribution matrix. represents a matrix consisting of real numbers, Represents a real number with width 1 and height d r Matrix of

[0025] Step 3-4: Convert the collected audio data into a time-frequency spectrum matrix, represented as a two-dimensional matrix Where A represents the time-frequency spectrum matrix, which can also be called the audio spectrum graph. represents a matrix consisting of real numbers, represents a matrix of width p and height q composed of real numbers, where p represents the frequency dimension and q represents the time dimension. The audio spectrogram A is calculated using the spectrum analysis technique of short-time Fourier transform (STFT);

[0026] Step 3-5: Input the spectrum matrix A into the graph convolutional neural network GCN model in the multimodal feature extraction module, and extract the time-frequency features of the audio through GCN. This processing process can be described as:

[0027]

[0028] in, is the node feature matrix of the lth layer, is the node feature matrix of the lth layer, l is the current number of layers of the node feature matrix, and at the initial moment, A' is the adjacency matrix of the audio data, which represents the local dependency of the audio data in the time-frequency diagram, A represents the time-frequency spectrum matrix, and D is the degree matrix. is the weight matrix of layer l. σ(·) is the ReLU activation function;

[0029] Step 3-6: After multiple layers of GCN convolution operations, the final audio data features extracted can be described as:

[0030]

[0031] where d A Represents the feature dimension of the extracted audio data, H A represents the node feature matrix, and A represents the time-frequency spectrum matrix. represents a matrix consisting of real numbers, Represents a real number with width 1 and height d AMatrix of

[0032] Step 3-7: Matrix concatenate the extracted temperature data features and audio data features to obtain the final multimodal feature vector. The concatenated result is the joint representation of the temperature data features and the audio data features.

[0033] In order to facilitate the feature fusion operation of multimodal information of mulberry pests and diseases, in the steps 3-7, the splicing operation of multimodal features is performed according to the following formula:

[0034]

[0035] The above formula represents the extracted temperature data feature H T and audio data features H A Perform matrix concatenation to obtain the final multimodal feature vector H multi Among them, H T Represents the temperature data characteristics, H A Represents the audio data features, H multi H represents the multimodal feature vector. T ‖H A Indicates horizontal concatenation of two feature vectors. is the node feature matrix of the temperature data features of the lth layer, is the node feature matrix of the temperature data features of the first layer, is the node feature matrix of the temperature data features of the second layer, It is the d T The node feature matrix of the temperature data features of the layer. T It represents the characteristic dimension of the extracted temperature data, and T represents the thermal imaging temperature distribution matrix. is the node feature matrix of the audio data features of the lth layer, is the node feature matrix of the audio data features at the first layer, is the node feature matrix of the audio data features at the second layer, It is the d A The node feature matrix of the audio data features of the layer. A Represents the feature dimension of the extracted audio data, and A represents the time-frequency spectrum matrix. The concatenated result H multi That is, the joint representation of temperature data features and audio data features.

[0036] In order to fuse the feature information of the image modality, thermal imaging modality and audio modality of mulberry tree pests and diseases into a joint feature vector, in step S4, cross-modal feature fusion is performed according to the following steps:

[0037] Step 5-1: The image modality decision vector H output by the image feature calculation module in the multimodal YOLO algorithm model img and the multimodal feature vector in the multimodal YOLO algorithm model output by the multimodal feature extraction module H in the multimodal YOLO algorithm model multi Input into the cross-modal feature fusion module in the multimodal YOLO algorithm model. img is the feature vector extracted from the image modality, H multi is the feature vector extracted from thermal imaging and audio modalities;

[0038] Step 5-2: Transform the image modality decision vector H img With the multimodal feature vector H multi Perform the splicing operation and provide the following formula to obtain the joint feature vector H concat :

[0039] H concat =[H img ‖H multi ]

[0040] Among them, H img is the feature vector extracted from the image modality, H multi is the feature vector extracted from thermal imaging and audio modalities, H concat is the joint eigenvector. img ‖H multi Indicates horizontal concatenation of two feature vectors.

[0041] Step 5-3: The concatenated feature vector H concat Input into the Boltzmann machine neural network model for nonlinear transformation. The Boltzmann machine neural network further processes the input vector through the hidden layer to enhance the representation ability of the feature. The calculation formula of the hidden layer of the Boltzmann machine neural network is:

[0042]

[0043] Where: H hidden is the hidden layer output after nonlinear transformation, represents a matrix consisting of real numbers, represents a matrix of real numbers with width 1 and height h, where h is the dimension of the hidden layer. W is the weight matrix of the neural network, b is the bias vector, and σ(·) is the ReLU activation function.

[0044] Step 5-4: Output decision vector

[0045] The output H of the hidden layer hiddenThe final one-dimensional decision vector H is generated through a fully connected layer of a Boltzmann machine neural network model. decision The process can be described as:

[0046]

[0047] Among them, H decision is the final decision vector, represents a matrix consisting of real numbers, represents a matrix of real numbers with a width of 1 and a height of N, where N represents the number of state categories. V is the weight matrix and c is the bias vector. softmax(·) is the softmax function used to convert the output into a probability distribution so that the sum of all elements is 1. decision In , each element represents the probability that the sample belongs to a specific state category. The decision vector H after softmax activation decision The value of the i-th dimension in is It means the probability that the current sample belongs to the i-th category, where i is the sample number pointer.

[0048] In order to intelligently identify the risk level of misclassification of mulberry pests and diseases, in step S5, the calculation of the identification deviation risk is performed according to the following steps:

[0049] Step 6-1: One-dimensional decision vector H output from the multimodal YOLO algorithm decision Calculate the maximum probability value p in this vector max ,Right now:

[0050]

[0051] Among them, H decision is the decision vector, the decision vector H after softmax activation decision The value of the i-th dimension in is It means the probability that the current sample belongs to the i-th category, i is the sample number pointer. N is the number of categories, p max is the maximum probability of the current sample among all categories. max(·) is the function for calculating the maximum value.

[0052] Step 6-2: Calculate the entropy value as the uncertainty of the decision, based on the one-dimensional decision vector H decision , calculate the entropy value H of sample classification to measure the uncertainty of decision. The entropy value calculation formula is:

[0053]

[0054] Where H represents the entropy value of the decision vector. decisionis the decision vector, the decision vector H after softmax activation decision The value of the i-th dimension in is It means the probability that the current sample belongs to the i-th category, i is the sample number pointer. N is the number of categories. The larger the entropy value H is, the higher the uncertainty of the decision.

[0055] Step 6-3: Calculate the risk of identification bias, which is based on the maximum probability value p max And entropy value H is calculated comprehensively. Specific identification bias risk R risk The calculation formula is:

[0056] R risk =α(1-p max )+βH

[0057] Where H is the uncertainty measure calculated from the entropy value of the decision vector, R risk is the identification bias risk value, α and β are weight coefficients used to control the impact of the maximum probability value and entropy value on the bias risk. max It is the maximum probability of the current sample among all categories.

[0058] Step 6-4: Compare the risk value with the preset threshold and calculate the identification bias risk R risk The preset threshold R set by the user threshold For comparison, R threshold It is a threshold entered by the user and is used to judge the reliability of the model prediction. risk is the risk value of identification bias. If R risk ≥R threshold , which means that the reliability of the recognition result is low and further self-supervised label generation is needed; otherwise, the model believes that the recognition result is reliable enough and directly proceeds to the next step of sample category decision;

[0059] In order to perform classification operation based on the decision vector obtained by feature extraction, in step S7, sample category decision is implemented according to the following formula:

[0060] Step 7-1: Check the output of the self-supervised label generation module. First, check whether the self-supervised label generation module has generated a new category label for the current input sample. If the self-supervised label generation module has a new label L generated for the current sample, new , then directly extract the new label as the decision result L of the sample final .

[0061] Among them, L new is the new label for the current sample output by the self-supervised label generation module, L final is the sample label of the final decision result;

[0062] Step 7-2: Extract the maximum probability of the decision vector. If the self-supervised label generation module does not generate a new label, extract the maximum probability of the decision vector from the one-dimensional decision vector H. decision Extract the maximum value p of the category probability max and its corresponding category label, satisfying:

[0063]

[0064] Among them, H decision is the decision vector, the decision vector H after softmax activation decision The value of the i-th dimension in is It means the probability that the current sample belongs to the i-th category, i is the sample number pointer. N is the number of categories, p max is the maximum probability of the current sample among all categories. max(·) is the maximum value function;

[0065] Step 7-3: Extract the category label L corresponding to the maximum probability max , and use it as the final category decision result L of the current sample final Among them, L max is the category label corresponding to the maximum probability, L final is the final decision label as the output;

[0066] Step 7-4: Output the decision result. final As the final decision output of the sample, L final is the final sample label, which is the recognition result of the category to which the sample belongs.

[0067] According to another aspect of one or more embodiments of the present disclosure, a computer-readable storage medium is also provided.

[0068] A computer-readable storage medium having a plurality of instructions stored therein, wherein the instructions are suitable for being loaded by a processor of a terminal device and executed by a self-supervised mulberry pest and disease identification method integrating a multimodal YOLO algorithm as described in any one of claims 1 to 7.

[0069] According to another aspect of one or more embodiments of the present disclosure, a terminal device is also provided.

[0070] The terminal device adopts an Internet terminal device, including a processor and a computer-readable storage medium, the processor is used to implement each instruction; the computer-readable storage medium is used to store multiple instructions, characterized in that the instructions are suitable for being loaded by the processor and executing the self-supervised mulberry pest and disease identification method integrating the multimodal YOLO algorithm as described in any one of claims 1-7.

[0071] According to another aspect of one or more embodiments of the present disclosure, a self-supervised mulberry pest and disease identification system integrating a multimodal YOLO algorithm is also provided.

[0072] A self-supervised mulberry pest and disease identification system integrating a multimodal YOLO algorithm, characterized in that the system comprises a self-supervised mulberry pest and disease identification method integrating a multimodal YOLO algorithm as described in any one of claims 1-7.

[0073] Compared with the prior art, the self-supervised mulberry pest and disease identification method and system integrating the multimodal YOLO algorithm provided by the present invention has the following beneficial effects:

[0074] The present invention provides a self-supervised mulberry pest identification method and system integrating a multimodal YOLO algorithm. The self-supervised mulberry pest identification method integrating a multimodal YOLO algorithm proposes a Multi-Modal-YOLO algorithm integrating a multimodal information processing technology, i.e., a multimodal YOLO algorithm, and combines audio information and thermal imaging information to further improve the accuracy of the identification results of pest types. A self-supervised learning framework driven by a multimodal large model is introduced. When the risk of identification deviation exceeds a preset threshold, the multimodal large model is called to generate a category label, and the model is given the ability to dynamically generate new labels for new types of pests and diseases, thereby effectively preventing the uneven distribution of training data or the misidentification caused by the lack of a certain category of data; the present invention improves the efficiency and accuracy of mulberry pest control by accurately identifying targets such as disease spots and pests on mulberry leaves, reduces the impact on sericulture production, and thus promotes production increase. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 A method flow chart of a preferred embodiment of the self-supervised mulberry tree disease and insect pest identification method and system integrating the multimodal YOLO algorithm provided by the present invention. DETAILED DESCRIPTION

[0076] The present invention will be further described below in conjunction with the accompanying drawings and implementation modes.

[0077] Please refer to Figure 1 ,in, Figure 1 A method flow chart of a preferred embodiment of the self-supervised mulberry tree disease and insect pest identification method and system integrating the multimodal YOLO algorithm provided by the present invention.

[0078] like Figure 1 As shown, a self-supervised mulberry pest and disease identification method integrating a multimodal YOLO algorithm includes:

[0079] S1: Information collection and model training, using sensors to obtain multimodal data of mulberry leaves, including RGB color images, thermal imaging temperature distribution data, and audio data. Using the constructed RGB color image dataset and its label information, the image feature calculation module in the constructed multimodal YOLO algorithm model is trained. The multimodal YOLO algorithm model consists of an image feature calculation module, a multimodal feature extraction module, a cross-modal feature fusion module, and a self-supervised label generation module;

[0080] In step S1, the multimodal YOLO algorithm model is constructed based on the following modules:

[0081] The constructed multimodal YOLO algorithm model includes an image feature calculation module, a multimodal feature extraction module, a cross-modal feature fusion module, and a self-supervised label generation module. Among them, the image feature calculation module in the multimodal YOLO algorithm model is a YOLOv9 model suitable for image classification tasks, used to extract feature information from sample image data; the multimodal feature extraction module is a graph convolutional neural network model suitable for feature extraction tasks, used to extract key features from multimodal information; the cross-modal feature fusion module is a Boltzmann machine neural network model suitable for prediction tasks, used to fuse the features of multimodal data such as RGB color images, thermal imaging temperature distribution data, and audio data; the self-supervised label generation module is a large multimodal model suitable for text generation tasks, used to process input image information to generate a text label for the current sample;

[0082] S2: extracting image features based on the YOLO algorithm, calling the image feature calculation module in the trained multimodal YOLO algorithm model, recognizing the collected RGB color image, and outputting the obtained image modality decision vector. The image feature calculation module in the multimodal YOLO algorithm model is a YOLOv9 model suitable for image classification tasks;

[0083] S3: Multimodal feature extraction: input the thermal imaging temperature distribution data and audio data into the multimodal feature extraction module in the multimodal YOLO algorithm model. The multimodal feature extraction module in the multimodal YOLO algorithm model is a graph convolutional neural network model suitable for feature extraction tasks. Feature extraction is performed through the multimodal feature extraction module in the multimodal YOLO algorithm model to generate a unified multimodal feature vector;

[0084] In step S3, multimodal feature extraction is performed according to the following steps:

[0085] Step 3-1: Represent the collected thermal imaging temperature distribution data as a two-dimensional matrix Where T represents the thermal imaging temperature distribution matrix, m represents the width of the thermal imaging data, and n represents the height of the thermal imaging data. represents a matrix consisting of real numbers, represents a matrix of width m and height n composed of real numbers, each element of which represents the temperature value of a certain position of a mulberry leaf;

[0086] Step 3-2: Input the thermal imaging temperature distribution matrix T into the multimodal feature extraction module and perform feature extraction through the graph convolutional neural network model. This feature extraction can be described as:

[0087]

[0088] in, is the node feature matrix of the lth layer, is the node feature matrix of the lth layer, l is the current number of layers of the node feature matrix, represents the initial state of the node feature matrix, T represents the thermal imaging temperature distribution matrix. A is the adjacency matrix, which represents the spatial structure of the temperature data, D is the degree matrix, and i and j represent the row vector pointer and column vector pointer respectively. is the weight matrix of the lth layer, σ(·) is the ReLU activation function;

[0089] Step 3-3: After the multi-layer convolution operation of the convolutional neural network, the final temperature data features extracted meet the following requirements:

[0090]

[0091] where d T Represents the characteristic dimension of the extracted temperature data, H T represents the node feature matrix, and T represents the thermal imaging temperature distribution matrix. represents a matrix consisting of real numbers, Represents a real number with width 1 and height d r Matrix of

[0092] Step 3-4: Convert the collected audio data into a time-frequency spectrum matrix, represented as a two-dimensional matrix Where A represents the time-frequency spectrum matrix, which can also be called the audio spectrum graph. represents a matrix consisting of real numbers, represents a matrix of width p and height q composed of real numbers, where p represents the frequency dimension and q represents the time dimension. The audio spectrogram A is calculated using the spectrum analysis technique of short-time Fourier transform (STFT);

[0093] Step 3-5: Input the spectrum matrix A into the graph convolutional neural network GCN model in the multimodal feature extraction module, and extract the time-frequency features of the audio through GCN. This processing process can be described as:

[0094]

[0095] in, is the node feature matrix of the lth layer, is the node feature matrix of the lth layer, l is the current number of layers of the node feature matrix, and at the initial moment, A' is the adjacency matrix of the audio data, which represents the local dependency of the audio data in the time-frequency diagram, A represents the time-frequency spectrum matrix, and D is the degree matrix. is the weight matrix of layer l. σ(·) is the ReLU activation function;

[0096] Step 3-6: After multiple layers of GCN convolution operations, the final audio data features extracted can be described as:

[0097]

[0098] where d A Represents the feature dimension of the extracted audio data, H A represents the node feature matrix, and A represents the time-frequency spectrum matrix. represents a matrix consisting of real numbers, Represents a real number with width 1 and height d A Matrix of

[0099] Step 3-7: Matrix concatenate the extracted temperature data features and audio data features to obtain the final multimodal feature vector. The concatenated result is the joint representation of the temperature data features and the audio data features.

[0100] In steps 3-7, the splicing operation of multimodal features is performed according to the following formula:

[0101]

[0102] The above formula represents the extracted temperature data feature H T and audio data features H A Perform matrix concatenation to obtain the final multimodal feature vector H multi Among them, H T Represents the temperature data characteristics, H A Represents the audio data features, H multi H represents the multimodal feature vector. T ‖H A Indicates horizontal concatenation of two feature vectors. is the node feature matrix of the temperature data features of the lth layer, is the node feature matrix of the temperature data features of the first layer, is the node feature matrix of the temperature data features of the second layer, It is the d T The node feature matrix of the temperature data features of the layer. T It represents the characteristic dimension of the extracted temperature data, and T represents the thermal imaging temperature distribution matrix. is the node feature matrix of the audio data features of the lth layer, is the node feature matrix of the audio data features at the first layer, is the node feature matrix of the audio data features at the second layer, It is the d A The node feature matrix of the audio data features of the layer. A Represents the feature dimension of the extracted audio data, and A represents the time-frequency spectrum matrix. The concatenated result H multi That is, the joint representation of temperature data features and audio data features.

[0103] S4: Cross-modal feature fusion, feature fusion of the image modality decision vector output by the image feature calculation module in the multimodal YOLO algorithm model and the multimodal feature vector output by the multimodal feature extraction module, and outputting a one-dimensional decision vector, wherein the cross-modal feature fusion module in the multimodal YOLO algorithm model is a Boltzmann machine neural network model, and the one-dimensional decision vector is a 1×N-dimensional array, where N represents the number of state categories, and the sum of all values ​​in the array is 1. The first value in the one-dimensional decision vector represents the probability that the current data sample belongs to the normal category, the second value represents the probability that the current data sample belongs to the mulberry shriveling disease category, and the third value represents the probability that the current data sample belongs to the mulberry blight disease category, and each number therein represents the probability of a state category;

[0104] In step S4, cross-modal feature fusion is performed according to the following steps:

[0105] Step 5-1: The image modality decision vector H output by the image feature calculation module in the multimodal YOLO algorithm model img and the multimodal feature vector in the multimodal YOLO algorithm model output by the multimodal feature extraction module H in the multimodal YOLO algorithm model multi Input into the cross-modal feature fusion module in the multimodal YOLO algorithm model. img is the feature vector extracted from the image modality, H multi is the feature vector extracted from thermal imaging and audio modalities;

[0106] Step 5-2: Transform the image modality decision vector H img With the multimodal feature vector H multi Perform the splicing operation and provide the following formula to obtain the joint feature vector H concat :

[0107] H concat =[H img ||H multi ]

[0108] Among them, H img is the feature vector extracted from the image modality, H multi is the feature vector extracted from thermal imaging and audio modalities, H concat is the joint eigenvector. img ||H multi Indicates horizontal concatenation of two feature vectors.

[0109] Step 5-3: The concatenated feature vector H concat Input into the Boltzmann machine neural network model for nonlinear transformation. The Boltzmann machine neural network further processes the input vector through the hidden layer to enhance the representation ability of the feature. The calculation formula of the hidden layer of the Boltzmann machine neural network is:

[0110]

[0111] Where: H hidden is the hidden layer output after nonlinear transformation, represents a matrix consisting of real numbers, represents a matrix of real numbers with width 1 and height h, where h is the dimension of the hidden layer. W is the weight matrix of the neural network, b is the bias vector, and σ(·) is the ReLU activation function.

[0112] Step 5-4: Output decision vector

[0113] The output H of the hidden layer hidden The final one-dimensional decision vector H is generated through a fully connected layer of a Boltzmann machine neural network model. decision The process can be described as:

[0114]

[0115] Among them, H decision is the final decision vector, represents a matrix consisting of real numbers, represents a matrix of real numbers with a width of 1 and a height of N, where N represents the number of state categories. V is the weight matrix and c is the bias vector. softmax(·) is the softmax function used to convert the output into a probability distribution so that the sum of all elements is 1. decision In , each element represents the probability that the sample belongs to a specific state category. The decision vector H after softmax activation decision The value of the i-th dimension in is It means the probability that the current sample belongs to the i-th category, where i is the sample number pointer.

[0116] S5: Identification bias risk calculation, calculating the identification bias risk of the one-dimensional decision vector output by the multimodal YOLO algorithm. When it exceeds the preset threshold input by the user, go to step S6; otherwise, go to step S7;

[0117] In step S5, the calculation of the identification deviation risk is performed according to the following steps:

[0118] Step 6-1: One-dimensional decision vector H output from the multimodal YOLO algorithm decision Calculate the maximum probability value p in this vector max ,Right now:

[0119]

[0120] Among them, H decision is the decision vector, the decision vector H after softmax activation decision The value of the i-th dimension in is It means the probability that the current sample belongs to the i-th category, i is the sample number pointer. N is the number of categories, p max is the maximum probability of the current sample among all categories. max(·) is the function for calculating the maximum value.

[0121] Step 6-2: Calculate the entropy value as the uncertainty of the decision, based on the one-dimensional decision vector H decision , calculate the entropy value H of sample classification to measure the uncertainty of decision. The entropy value calculation formula is:

[0122]

[0123] Where H represents the entropy value of the decision vector. decision is the decision vector, the decision vector H after softmax activation decision The value of the i-th dimension in is It means the probability that the current sample belongs to the i-th category, i is the sample number pointer. N is the number of categories. The larger the entropy value H is, the higher the uncertainty of the decision.

[0124] Step 6-3: Calculate the risk of identification bias, which is based on the maximum probability value p max And entropy value H is calculated comprehensively. Specific identification bias risk R risk The calculation formula is:

[0125] R risk =α(1-p max )+βH

[0126] Where H is the uncertainty measure calculated from the entropy value of the decision vector, R risk is the identification bias risk value, α and β are weight coefficients used to control the impact of the maximum probability value and entropy value on the bias risk. max It is the maximum probability of the current sample among all categories.

[0127] Step 6-4: Compare the risk value with the preset threshold and calculate the identification bias risk R risk The preset threshold R set by the user threshold For comparison, R threshold It is a threshold entered by the user and is used to judge the reliability of the model prediction. risk is the risk value of identification bias. If R risk ≥R threshold , which means that the reliability of the recognition result is low and further self-supervised label generation is needed; otherwise, the model believes that the recognition result is reliable enough and directly proceeds to the next step of sample category decision;

[0128] S6: self-supervised label generation, calling the self-supervised label generation module in the multimodal YOLO algorithm model to generate category labels, giving the model the ability to dynamically generate new labels for new types of pests and diseases, thereby effectively preventing misidentification caused by uneven distribution of training data or missing data of a certain category. The self-supervised label generation module in the multimodal YOLO algorithm model is a large multimodal model used to process input image information to generate a text label for the current sample;

[0129] S7: Sample category decision: if the self-supervised label generation module in the multimodal YOLO algorithm model has a new label generated for the current sample, the new label is extracted; otherwise, the maximum value of the category probability is extracted from the one-dimensional decision vector, and the sample label corresponding to the value is extracted. The extracted sample label is output as the decision result.

[0130] In step S7, sample category decision is implemented according to the following formula:

[0131] Step 7-1: Check the output of the self-supervised label generation module. First, check whether the self-supervised label generation module has generated a new category label for the current input sample. If the self-supervised label generation module has a new label L generated for the current sample, new , then directly extract the new label as the decision result L of the sample final .

[0132] Among them, L new is the new label for the current sample output by the self-supervised label generation module, L final is the sample label of the final decision result;

[0133] Step 7-2: Extract the maximum probability of the decision vector. If the self-supervised label generation module does not generate a new label, extract the maximum probability of the decision vector from the one-dimensional decision vector H. decision Extract the maximum value p of the category probability max and its corresponding category label, satisfying:

[0134]

[0135] Among them, H decision is the decision vector, the decision vector H after softmax activation decision The value of the i-th dimension in is It means the probability that the current sample belongs to the i-th category, i is the sample number pointer. N is the number of categories, p max is the maximum probability of the current sample among all categories. max(·) is the maximum value function;

[0136] Step 7-3: Extract the category label L corresponding to the maximum probability max , and use it as the final category decision result L of the current sample final Among them, L max is the category label corresponding to the maximum probability, L final is the final decision label as the output;

[0137] Step 7-4: Output the decision result. final As the final decision output of the sample, L final is the final sample label, which is the recognition result of the category to which the sample belongs.

[0138] In one or more embodiments of the present disclosure, the recognition accuracy results of the self-supervised mulberry pest and disease identification method integrating the multimodal YOLO algorithm are shown in Table 1. The evaluation results are automated evaluations performed on the local sericulture pest and disease identification dataset, including four dimensions: test set accuracy, five-fold classification accuracy, new category accuracy, and comprehensive level. The values ​​of each dimension are distributed between 0% and 100%. Among them, the test set accuracy refers to the classification accuracy value obtained by randomly splitting the local sericulture pest and disease identification dataset into a training set accounting for 80% and a test set accounting for 20%, using the training set data to train the model, and then using the test set to evaluate the model. The five-fold classification accuracy refers to the classification accuracy value obtained by randomly splitting the local sericulture pest and disease identification dataset into five sub-datasets, Dataset-A, Dataset-B, Dataset-C, Dataset-D, and Dataset-E, each accounting for 20%, and calculating Dataset-A as the test set and the rest as the training set. A , the classification accuracy value obtained by using Dataset-B as the test set and the rest as the training set Result B , the classification accuracy value Result obtained by using Dataset-C as the test set and the rest as the training set C , the classification accuracy value Result obtained by using Dataset-D as the test set and the rest as the training set D , the classification accuracy value Result obtained by taking Dataset-E as the test set and the rest as the training set E , and then calculate the classification accuracy value Result obtained five times A ,Result B ,Result C ,Result D ,Result E The average of the five-fold classification accuracy is used as the final five-fold classification accuracy, thereby eliminating the impact of randomness on the classification results during the random split of the training set and the test set. The new category accuracy refers to the classification accuracy of sample data belonging to unlabeled new categories in the local silkworm and mulberry pest identification dataset. The comprehensive level refers to the average of the three indicators of test set accuracy, five-fold classification accuracy, and new category accuracy.

[0139] Table 1 Recognition accuracy results of the self-supervised mulberry pest and disease recognition method fused with the multimodal YOLO algorithm in one or more embodiments of the present disclosure

[0140]

[0141] According to another aspect of one or more embodiments of the present disclosure, a computer-readable storage medium is also provided.

[0142] A computer-readable storage medium having a plurality of instructions stored therein, wherein the instructions are suitable for being loaded by a processor of a terminal device and executed by a self-supervised mulberry pest and disease identification method integrating a multimodal YOLO algorithm as described in any one of claims 1 to 7.

[0143] According to another aspect of one or more embodiments of the present disclosure, a terminal device is also provided.

[0144] The terminal device adopts an Internet terminal device, including a processor and a computer-readable storage medium, the processor is used to implement each instruction; the computer-readable storage medium is used to store multiple instructions, characterized in that the instructions are suitable for being loaded by the processor and executing the self-supervised mulberry pest and disease identification method integrating the multimodal YOLO algorithm as described in any one of claims 1-7.

[0145] According to another aspect of one or more embodiments of the present disclosure, a self-supervised mulberry pest and disease identification system integrating a multimodal YOLO algorithm is also provided.

[0146] A self-supervised mulberry pest and disease identification system integrating a multimodal YOLO algorithm, characterized in that the system comprises a self-supervised mulberry pest and disease identification method integrating a multimodal YOLO algorithm as described in any one of claims 1-7.

[0147] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0148] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, the storage medium including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0149] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

Claims

1. A self-supervised mulberry pest and disease identification method integrating the multimodal YOLO algorithm is characterized in that: The method comprises the following steps: S1: Information collection and model training, obtaining multimodal data of mulberry leaves through sensors, including RGB color images, thermal imaging temperature distribution data, and audio data, and using the constructed RGB color image data set and its label information to train the image feature calculation module in the constructed multimodal YOLO algorithm model. The multimodal YOLO algorithm model consists of an image feature calculation module, a multimodal feature extraction module, a cross-modal feature fusion module, and a self-supervised label generation module; S2: extracting image features based on the YOLO algorithm, calling the image feature calculation module in the trained multimodal YOLO algorithm model, recognizing the collected RGB color image, and outputting the obtained image modality decision vector. The image feature calculation module in the multimodal YOLO algorithm model is a YOLOv9 model suitable for image classification tasks; S3: multimodal feature extraction, inputting the thermal imaging temperature distribution data and the audio data into a multimodal feature extraction module in the multimodal YOLO algorithm model, wherein the multimodal feature extraction module in the multimodal YOLO algorithm model is a graph convolutional neural network model suitable for feature extraction tasks, and performing feature extraction through the multimodal feature extraction module in the multimodal YOLO algorithm model to generate a unified multimodal feature vector; S4: cross-modal feature fusion, feature fusion of the image modal decision vector output by the image feature calculation module in the multimodal YOLO algorithm model and the multimodal feature vector output by the multimodal feature extraction module, and outputting a one-dimensional decision vector, wherein the cross-modal feature fusion module in the multimodal YOLO algorithm model is a Boltzmann machine neural network model, and the one-dimensional decision vector is a 1×N-dimensional array, wherein N represents the number of state categories, and the sum of all values ​​in the array is 1, and the first value in the one-dimensional decision vector represents the probability that the current data sample belongs to the normal category, the second value represents the probability that the current data sample belongs to the mulberry shriveling disease category, and the third value represents the probability that the current data sample belongs to the mulberry blight disease category, and each number therein represents the probability of a state category; S5: Identification bias risk calculation, calculating the identification bias risk of the one-dimensional decision vector output by the multimodal YOLO algorithm. When it exceeds the preset threshold input by the user, go to step S6; otherwise, go to step S7; S6: self-supervised label generation, calling the self-supervised label generation module in the multimodal YOLO algorithm model to generate category labels, giving the model the ability to dynamically generate new labels for new types of pests and diseases, thereby effectively preventing misidentification caused by uneven distribution of training data or missing data of a certain category. The self-supervised label generation module in the multimodal YOLO algorithm model is a large multimodal model used to process input image information to generate a text label for the current sample; S7: Sample category decision. If the self-supervised label generation module in the multimodal YOLO algorithm model has a new label generated for the current sample, the new label is extracted; otherwise, the maximum value of the category probability is extracted from the one-dimensional decision vector, and the sample label corresponding to the value is extracted, and the extracted sample label is output as the decision result.

2. The self-supervised mulberry tree pest and disease identification method integrating the multimodal YOLO algorithm according to claim 1, characterized in that: In the step S1, the constructed multimodal YOLO algorithm model includes an image feature calculation module, a multimodal feature extraction module, a cross-modal feature fusion module, and a self-supervised label generation module, wherein the image feature calculation module in the multimodal YOLO algorithm model is a YOLOv9 model suitable for image classification tasks, used to extract feature information in sample image data; the multimodal feature extraction module is a graph convolutional neural network model suitable for feature extraction tasks, used to extract key features from multimodal information; the cross-modal feature fusion module is a Boltzmann machine neural network model suitable for prediction tasks, used to fuse features of multimodal data such as RGB color images, thermal imaging temperature distribution data, and audio data; the self-supervised label generation module is a large multimodal model suitable for text generation tasks, used to process input image information to generate a text label for the current sample.

3. The self-supervised mulberry pest identification method integrating the multimodal YOLO algorithm according to claim 1, characterized in that: In step S3, multimodal feature extraction is performed according to the following steps: Step 3-1: Represent the collected thermal imaging temperature distribution data as a two-dimensional matrix Where T represents the thermal imaging temperature distribution matrix, m represents the width of the thermal imaging data, and n represents the height of the thermal imaging data. represents a matrix consisting of real numbers, represents a matrix of width m and height n composed of real numbers, each element of which represents the temperature value of a certain position of a mulberry leaf; Step 3-2: Input the thermal imaging temperature distribution matrix T into the multimodal feature extraction module and perform feature extraction through the graph convolutional neural network model. This feature extraction can be described as: in, is the node feature matrix of the lth layer, is the node feature matrix of the lth layer, l is the current number of layers of the node feature matrix, represents the initial state of the node feature matrix, T represents the thermal imaging temperature distribution matrix, A is the adjacency matrix, which represents the spatial structure of the temperature data, D is the degree matrix, i and j represent the row vector pointer and column vector pointer respectively, is the weight matrix of the lth layer, σ(·) is the ReLU activation function; Step 3-3: After the multi-layer convolution operation of the convolutional neural network, the final temperature data features extracted meet the following requirements: where d T Represents the characteristic dimension of the extracted temperature data, H T represents the node feature matrix, T represents the thermal imaging temperature distribution matrix, represents a matrix consisting of real numbers, Represents a real number with width 1 and height d r Matrix of Step 3-4: Convert the collected audio data into a time-frequency spectrum matrix, represented as a two-dimensional matrix Where A represents the time-frequency spectrum matrix, which can also be called the audio spectrum graph. represents a matrix consisting of real numbers, represents a matrix of width p and height q composed of real numbers, where p represents the frequency dimension and q represents the time dimension, wherein the audio spectrogram A is calculated by the spectrum analysis technique of short-time Fourier transform STFT; Step 3-5: Input the spectrum matrix A into the graph convolutional neural network GCN model in the multimodal feature extraction module, and extract the time-frequency features of the audio through GCN. This processing process can be described as: in, is the node feature matrix of the lth layer, is the node feature matrix of the lth layer, l is the current number of layers of the node feature matrix, and at the initial moment, A' is the adjacency matrix of the audio data, which represents the local dependency of the audio data in the time-frequency diagram. A represents the time-frequency spectrum matrix, and D is the degree matrix. is the weight matrix of the lth layer, σ(·) is the ReLU activation function; Step 3-6: After multiple layers of GCN convolution operations, the final audio data features extracted can be described as: where d A Represents the feature dimension of the extracted audio data, H A represents the node feature matrix, A represents the time-frequency spectrum matrix, represents a matrix consisting of real numbers, Represents a real number with width 1 and height d A Matrix of Step 3-7: Matrix concatenate the extracted temperature data features and audio data features to obtain the final multimodal feature vector. The concatenated result is the joint representation of the temperature data features and the audio data features.

4. The self-supervised mulberry pest identification method integrating the multimodal YOLO algorithm according to claim 1, characterized in that: In the steps 3-7, the splicing operation of the multimodal features is performed according to the following formula: The above formula represents the extracted temperature data feature H T and audio data features H A Perform matrix concatenation to obtain the final multimodal feature vector H multi , where H T Represents the temperature data characteristics, H A Represents the audio data features, H multi represents the multimodal feature vector, H T ||H A Indicates the horizontal concatenation of two feature vectors. is the node feature matrix of the temperature data features of the lth layer, is the node feature matrix of the temperature data features of the first layer, is the node feature matrix of the temperature data features of the second layer, It is the d T The node feature matrix of the temperature data features of the layer, d T represents the characteristic dimension of the extracted temperature data, T represents the thermal imaging temperature distribution matrix, is the node feature matrix of the audio data features of the lth layer, is the node feature matrix of the audio data features at the first layer, is the node feature matrix of the audio data features at the second layer, It is the d A The node feature matrix of the audio data features of the layer, d A represents the feature dimension of the extracted audio data, A represents the time-frequency spectrum matrix, and the concatenated result H multi That is, the joint representation of temperature data features and audio data features.

5. The self-supervised mulberry pest identification method integrating the multimodal YOLO algorithm according to claim 1, characterized in that: In step S4, cross-modal feature fusion is performed according to the following steps: Step 5-1: The image modality decision vector H output by the image feature calculation module in the multimodal YOLO algorithm model img and the multimodal feature vector in the multimodal YOLO algorithm model output by the multimodal feature extraction module H in the multimodal YOLO algorithm model multi Input into the cross-modal feature fusion module in the multimodal YOLO algorithm model, where H img is the feature vector extracted from the image modality, H multi is the feature vector extracted from thermal imaging and audio modalities; Step 5-2: Transform the image modality decision vector H img With the multimodal feature vector H multi Perform the splicing operation and provide the following formula to obtain the joint feature vector H concat : H concat =[H img ||H multi ] Among them, H img is the feature vector extracted from the image modality, H multi is the feature vector extracted from thermal imaging and audio modalities, H concat is the joint eigenvector, H img ‖H multi Indicates the horizontal concatenation of two feature vectors. Step 5-3: The concatenated feature vector H concat The input is input into the Boltzmann machine neural network model for nonlinear transformation. The Boltzmann machine neural network further processes the input vector through the hidden layer to enhance the representation ability of the feature. The calculation formula of the hidden layer of the Boltzmann machine neural network is: Where: H hidden is the hidden layer output after nonlinear transformation, represents a matrix consisting of real numbers, represents a matrix of real numbers with a width of 1 and a height of h, where h is the dimension of the hidden layer, W is the weight matrix of the neural network, b is the bias vector, σ(·) is the ReLU activation function, Step 5-4: Output decision vector The output H of the hidden layer hidden The final one-dimensional decision vector H is generated through a fully connected layer of a Boltzmann machine neural network model. decision , this process can be described as: Among them, H decision is the final decision vector, represents a matrix consisting of real numbers, represents a matrix of width 1 and height N composed of real numbers, where N represents the number of state categories, V is the weight matrix, c is the bias vector, and softmax(·) is the softmax function used to convert the output into a probability distribution so that the sum of all elements is 1. decision In , each element represents the probability that the sample belongs to a specific state category, and the decision vector H after softmax activation decision The value of the i-th dimension in is It means the probability that the current sample belongs to the i-th category, where i is the sample number pointer.

6. The self-supervised mulberry pest identification method integrating the multimodal YOLO algorithm according to claim 1, characterized in that: In step S5, the calculation of the identification deviation risk is performed according to the following steps: Step 6-1: One-dimensional decision vector H output from the multimodal YOLO algorithm decision Calculate the maximum probability value p in this vector max ,Right now: Among them, H decision is the decision vector, the decision vector H after softmax activation decision The value of the i-th dimension in is It means the probability that the current sample belongs to the i-th category, i is the sample number pointer, N is the number of categories, and p max is the maximum probability of the current sample among all categories, max(·) is the maximum value function. Step 6-2: Calculate the entropy value as the uncertainty of the decision, based on the one-dimensional decision vector H decision , calculate the entropy value H of sample classification to measure the uncertainty of decision making. The entropy value calculation formula is: Among them, H represents the entropy value of the decision vector, H decision is the decision vector, the decision vector H after softmax activation decision The value of the i-th dimension in is It means the probability that the current sample belongs to the i-th category, i is the sample number pointer, N is the number of categories, and the larger the entropy value H is, the higher the uncertainty of the decision. Step 6-3: Calculate the risk of identification bias, which is based on the maximum probability value p max The specific identification bias risk R is calculated by comprehensive calculation with entropy value H. risk The calculation formula is: R risk =α(1-p max )+βH Where H is the uncertainty measure calculated from the entropy value of the decision vector, R risk is the identification bias risk value, α and β are weight coefficients used to control the impact of the maximum probability value and entropy value on the bias risk, p max is the maximum probability of the current sample among all categories. Step 6-4: Compare the risk value with the preset threshold and calculate the identification bias risk R risk The preset threshold R set by the user threshold For comparison, R threshold is the threshold entered by the user to determine the reliability of the model prediction. risk is the risk value of identification bias, if R risk ≥R threshold , which means that the reliability of the recognition result is low and further self-supervised label generation is needed; otherwise, the model believes that the recognition result is reliable enough and directly proceeds to the next step of sample category decision.

7. The self-supervised mulberry pest identification method integrating the multimodal YOLO algorithm according to claim 1, characterized in that: In step S7, sample category decision is implemented according to the following formula: Step 7-1: Check the output of the self-supervised label generation module. First, check whether the self-supervised label generation module has generated a new category label for the current input sample. If the self-supervised label generation module has a new label L generated for the current sample, new , then directly extract the new label as the decision result L of the sample final , Among them, L new is the new label for the current sample output by the self-supervised label generation module, L final is the sample label of the final decision result; Step 7-2: Extract the maximum probability of the decision vector. If the self-supervised label generation module does not generate a new label, extract the maximum probability of the decision vector from the one-dimensional decision vector H. decision Extract the maximum value p of the category probability max and its corresponding category label, satisfying: Among them, H decision is the decision vector, the decision vector H after softmax activation decision The value of the i-th dimension in is It means the probability that the current sample belongs to the i-th category, i is the sample number pointer, N is the number of categories, and p max is the maximum probability of the current sample among all categories, and max(·) is the function for calculating the maximum value; Step 7-3: Extract the category label L corresponding to the maximum probability max , and use it as the final category decision result L of the current sample final , where L max is the category label corresponding to the maximum probability, L final is the final decision label as the output; Step 7-4: Output the decision result and convert the L obtained in the previous step final As the final decision output of the sample, L final is the final sample label, which is the recognition result of the category to which the sample belongs.

8. A computer-readable storage medium having a plurality of instructions stored therein, characterized in that: The instructions are suitable for being loaded by a processor of a terminal device and executing the self-supervised mulberry pest and disease identification method integrating a multimodal YOLO algorithm as described in any one of claims 1-7.

9. A terminal device, using an Internet terminal device, comprising a processor and a computer-readable storage medium, wherein the processor is used to implement each instruction; and the computer-readable storage medium is used to store multiple instructions, characterized in that: The instructions are suitable for being loaded by a processor and executing the self-supervised mulberry pest and disease identification method integrating the multimodal YOLO algorithm as described in any one of claims 1-7.

10. A self-supervised mulberry pest and disease identification system integrating the multimodal YOLO algorithm is characterized by: The system includes a self-supervised mulberry pest and disease identification method integrating a multimodal YOLO algorithm as described in any one of claims 1-7.

Images