Tea insect pest recognition and prediction method fusing multi-modal information

Abstract

The invention provides a tea insect pest recognition and prediction method fused with multi-modal information in the technical field of tea insect pest recognition and prediction, and the method comprises the steps: S1, collecting a tea image and environment data, and constructing an image data set and an environment data set; s2, dividing the image data set into a training set, a test set and a verification set; s3, creating a tea insect pest recognition model based on a YOLOv8n network; s4, training the tea insect pest recognition model through the training set, testing the trained tea insect pest recognition model through the test set, and verifying the tested tea insect pest recognition model through the verification set; fitting a pest quantity prediction function through the environment data set; and S5, performing tea insect pest recognition through the verified tea insect pest recognition model, and predicting the number of tea insect pests through the fitted insect pest number prediction function. The method has the advantages that the tea pest recognition accuracy is greatly improved, and the number of pests is predicted.

Description

Technical Field

[0001] The present invention relates to the technical field of tea pest identification and prediction, and in particular to a tea pest identification and prediction method integrating multi-modal information. Background Art

[0002] The favorable ecological environment of tea trees easily leads to the breeding of complex and diverse pests in tea leaves. Pests are directly related to the yield and quality of tea, so pests need to be prevented and controlled. In order to carry out targeted prevention and control of pests, pests need to be identified.

[0003] With the development of deep learning, Internet of Things, big data and other technologies, the field of intelligent prevention and control of agricultural pests has developed rapidly. At present, the identification of tea pests is mainly carried out by analyzing the pest features (image features) in tea images, but the existing recognition algorithms are difficult to process high-resolution images, have limitations in extracting pest features, and have problems such as poor adaptability to multi-scale changes and easy loss of important information, resulting in poor accuracy in identifying tea pests; and traditionally, it is impossible to predict the growth of pests, that is, it is impossible to predict the number of pests in the future, and thus it is impossible to effectively adjust prevention and control measures. For example, if it can be predicted that the pests will grow rapidly, the concentration of the insecticide can be appropriately increased at this time.

[0004] Therefore, how to provide a tea pest identification and prediction method that integrates multimodal information to improve the accuracy of tea pest identification and predict the number of pests has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a tea pest identification and prediction method integrating multimodal information, so as to improve the accuracy of tea pest identification and predict the number of pests.

[0006] The present invention is implemented as follows: a method for identifying and predicting tea pests by integrating multimodal information, comprising the following steps:

[0007] Step S1, collecting a large number of tea images containing pests of different types and numbers, and historical environmental data corresponding to each of the tea images, constructing an image dataset after preprocessing each of the tea images, and constructing an environmental dataset based on each of the historical environmental data and the tea images;

[0008] Step S2, dividing the image data set into a training set, a test set and a validation set;

[0009] Step S3, creating a tea pest recognition model based on the input module, feature extraction module, neck module and head module of the YOLOv8n network, and setting the loss function of the tea pest recognition model to the WIoUv3 function;

[0010] The feature extraction module is composed of SCConv units; the neck module is composed of EMA units; the head module is composed of bounding box regression layers and classification layers;

[0011] Step S4, training the tea pest identification model with the training set, testing the trained tea pest identification model with the test set, and verifying the tested tea pest identification model with the verification set; and obtaining a pest quantity prediction function by fitting the environmental data set;

[0012] Step S5: identifying tea pests using the verified tea pest identification model, and predicting the number of tea pests using the fitted pest number prediction function.

[0013] Furthermore, the step S1 is specifically as follows:

[0014] Collect a large number of tea images containing different types and numbers of pests and the environmental data corresponding to each of the tea images, annotate each of the tea images with pest boundary boxes, pest types, and pest numbers through Labelimg, perform brightness adjustment, contrast adjustment, or random rotation sample expansion operations on each of the annotated tea images, and then complete the preprocessing of each of the tea images, and construct a data set based on the preprocessed tea images;

[0015] Constructing an environmental data set based on the historical environmental data and the pest types and pest quantities of the corresponding tea leaves;

[0016] The historical environmental data includes at least relative humidity, rainfall probability, air pressure, snowfall, daily rainfall, monthly rainfall, wind direction, wind level, wind speed, sunrise time, sunset time, real-time temperature, perceived temperature, ultraviolet intensity, visibility, light radiation, sunshine duration, night duration, CO2 concentration, PM2.5 value and PM10 value.

[0017] Furthermore, the step S2 is specifically as follows:

[0018] The image dataset is divided into a training set, a test set, and a validation set based on a ratio of 6:2:2.

[0019] Furthermore, in step S3, the input module is used to input tea images; the feature extraction module is used to extract pest features from the tea images; the neck module is used to fuse the extracted pest features; and the head module is used to output pest identification results.

[0020] Furthermore, in step S3, the formula of the WIoUv3 function is:

[0021] L WIoUv3=rL WIoUv1 ;

[0022] r=β / δα β-δ ;

[0023]

[0024] L WIoUv1 =R WIoU L IoU ;

[0025]

[0026] Among them, L WIoUv3 represents the loss value of WIoUv3 function; r represents the non-monotonic focusing coefficient; L WIoUv1 represents the loss value of the WIoUv1 function; β represents the outlier coefficient; α and δ both represent hyperparameters that control the characteristics of the model; Indicates L WIoUv1 The monotonic focusing coefficient of represents the sliding average with momentum m; R WIoU ∈[1,e), represents a normal anchor box with enhanced quality; L IoU represents a high-quality anchor box whose quality has been reduced; (x, y) represents the coordinates of the upper left corner of the ordinary anchor box; (x gt ,y gt ) represents the coordinates of the upper left corner of the target box; W g Indicates the width of the pest boundary box; H g Indicates the height of the pest bounding box; represents the stable convergence factor.

[0027] Further, in step S3, the SCConv unit is composed of an SRU subunit and a CRU subunit; the SRU subunit is connected to the CRU subunit;

[0028] The SRU subunit is used to suppress the spatial redundancy of the tea leaf image; and the CRU subunit is used to suppress the channel redundancy of the tea leaf image.

[0029] Furthermore, in step S3, the EMA unit is composed of a 1×1 convolution subunit, a 3×3 convolution subunit and a cross-space learning subunit.

[0030] Furthermore, in step S3, the bounding box regression layer is used to predict the location of the pests and output a predicted pest bounding box;

[0031] The classification layer is used for prediction of pest types.

[0032] Furthermore, the step S4 is specifically as follows:

[0033] The tea pest identification model is trained by the training set, and the hyperparameters of the tea pest identification model including at least a random deactivation rate, a weight decay rate, and a learning rate are continuously optimized during the training process until the loss value of the loss function is less than a preset loss threshold;

[0034] The trained tea pest identification model is tested by the test set to determine whether the identification accuracy is greater than a preset accuracy threshold. If not, the training set is expanded to continue training; if yes, then:

[0035] Calculating the accuracy, recall rate, F1 score, average precision and mean of average precision of the tested tea pest identification model through the verification set, and verifying the performance of the tea pest identification model through the accuracy, recall rate, F1 score, average precision and mean of average precision;

[0036] The number of pests in the environmental data set is taken as the dependent variable, the pest type and historical environmental data in the environmental data set are taken as independent variables, each of the dependent variables and independent variables is fitted to obtain the weight coefficient corresponding to each of the independent variables, and then the pest number prediction function is obtained.

[0037] Furthermore, the step S5 is specifically as follows:

[0038] The tea image to be identified and the real-time environmental data corresponding to the tea image to be identified are obtained. After the tea image to be identified is preprocessed for image enhancement, the tea image to be identified is input into a verified tea pest identification model, and a tea pest identification result containing at least a pest boundary box, pest type, and identification time is output; the real-time environmental data is input into a pest quantity prediction function, and a pest quantity prediction curve is output.

[0039] The advantages of the present invention are:

[0040] 1. By collecting a large number of tea images containing different types and numbers of pests, each tea image is preprocessed to construct a data set, and the data set is divided into a training set, a test set and a validation set; then a tea pest recognition model is created based on the input module, feature extraction module, neck module and head module of the YOLOv8n network, and the loss function of the tea pest recognition model is set to the WIoUv3 function; the feature extraction module is composed of SCConv units; the neck module is provided with an EMA unit; the head module is provided with a bounding box regression layer and a classification layer; then the tea pest recognition model is trained with the training set, the trained tea pest recognition model is tested with the test set, the tested tea pest recognition model is verified with the validation set, and finally the tea pest recognition model is used to recognize tea pests; that is, the traditional CIoU function is replaced by the WIoUv3 function, and a non-monotonic focusing system is introduced. The number is used to more accurately describe the boundary difference between the predicted box (predicted pest bounding box) and the true box (real pest bounding box), improve the detection effect of small targets, optimize the training process and enhance the generalization ability of the tea pest recognition model; the network structure of the traditional feature extraction module is improved through the SCConv unit, the spatial information and channel information are enriched, and redundant features (spatial redundancy, channel redundancy) are suppressed to maintain high accuracy and reduce the number of parameters of the tea pest recognition model; by introducing the EMA unit in the neck module, different feature semantics are effectively learned, and multi-scale pixel-level attention of the global context is realized, so that the tea pest recognition model pays more attention to the target area (pests), and improves the recognition accuracy of the tea pest recognition model for small targets, multiple targets and interference targets, that is, the tea pest recognition model created by the improved YOLOv8n network is used for tea pest recognition, which ultimately greatly improves the accuracy of tea pest recognition.

[0041] 2. By collecting a large number of tea images and the historical environmental data corresponding to each tea image, an environmental data set is constructed based on the historical environmental data and the pest types and pest quantities marked in the corresponding tea images; the pest quantity in the environmental data set is used as the dependent variable, and the pest type and historical environmental data in the environmental data set are used as independent variables. Each dependent variable and independent variable is fitted to obtain the weight coefficient corresponding to each variable, and then a pest quantity prediction function is obtained. Subsequently, the pest quantity prediction function can be used to predict the pest quantity, that is, to predict the pest quantity based on environmental data.

[0042] 3. Tea pests are identified through the tea pest identification model, and the growth number of tea pests is predicted through the pest quantity prediction function. It integrates the multimodal information of tea images and environmental data, and can carry out targeted prevention and control of different pests in different environments, effectively improving the pest control effect.

[0043] 4. By performing sample expansion operations such as brightness adjustment, contrast adjustment or random rotation on the labeled tea images, sufficient sample size can effectively prevent overfitting of the tea pest recognition model training, thereby effectively improving the generalization ability of the tea pest recognition model.

[0044] 5. By continuously optimizing the hyperparameters including at least the random inactivation rate, weight decay rate and learning rate during the training process of the tea pest identification model, the recognition accuracy of the trained tea pest identification model is tested through the test set, and the accuracy, recall rate, F1 score, average precision and average precision mean of the tested tea pest identification model are calculated through the validation set to verify the performance of the tea pest identification model. That is, multi-dimensional optimization and verification are performed during the training process of the tea pest identification model, thereby greatly improving the accuracy of tea pest identification.

[0045] 6. After preprocessing the tea image for image enhancement, the image is input into the tea pest recognition model to identify tea pests, thereby further improving the accuracy of tea pest recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0047] Figure 1 The present invention is a flow chart of a method for identifying and predicting tea pests by integrating multimodal information.

[0048] Figure 2 It is a structural schematic diagram of the tea pest identification model of the present invention.

[0049] Figure 3 It is a schematic diagram of the structure of the SCConv unit of the present invention.

[0050] Figure 4 It is a schematic diagram of the structure of the SRU subunit of the present invention.

[0051] Figure 5 It is a structural schematic diagram of the CRU subunit of the present invention.

[0052] Figure 6 It is a schematic diagram of the structure of the EMA unit of the present invention. DETAILED DESCRIPTION

[0053] The technical solution in the embodiment of the present application has the following general idea: tea pest recognition is performed by using a tea pest recognition model created by an improved YOLOv8n network. The YOLOv8n network replaces the traditional CIoU function with the WIoUv3 function, introduces a non-monotonic focusing coefficient to more accurately describe the boundary difference between the predicted box and the real box, improves the detection effect of small targets, optimizes the training process and enhances the generalization ability of the tea pest recognition model; improves the network structure of the traditional feature extraction module through the SCConv unit, enriches the spatial information and channel information, suppresses redundant features, so as to maintain a high accuracy and reduce the tea pest recognition model. Parameter quantity; the EMA unit is introduced in the neck module to effectively learn different feature semantics and realize multi-scale pixel-level attention of the global context, so that the tea pest recognition model pays more attention to the target area, and improves the recognition accuracy of the tea pest recognition model for small targets, multiple targets and interference targets, thereby improving the accuracy of tea pest recognition; by taking the number of pests in the environmental dataset as the dependent variable, and the pest type in the environmental dataset and the historical environmental data as the independent variables, each dependent variable and independent variable are fitted to obtain the weight coefficient corresponding to each variable, and then the pest quantity prediction function is obtained, so as to realize the prediction of the pest quantity using the pest quantity prediction function.

[0054] Please refer to Figures 1 to 6 As shown, a preferred embodiment of a tea pest identification and prediction method integrating multimodal information of the present invention comprises the following steps:

[0055] Step S1, collecting a large number of tea images containing pests of different types and numbers, and historical environmental data corresponding to each of the tea images, constructing an image dataset after preprocessing each of the tea images, and constructing an environmental dataset based on each of the historical environmental data and the tea images;

[0056] Step S2, dividing the image data set into a training set, a test set and a validation set;

[0057] Step S3, creating a tea pest recognition model based on the input module (Input), feature extraction module (Backbone), neck module (Neck) and head module (Head) of the YOLOv8n (YOLOv8 Nano) network, and setting the loss function of the tea pest recognition model to the WIoUv3 function;

[0058] The feature extraction module is provided with a SCConv unit (Spatial and Channel Reconstruction Convolution for Feature Redundancy); the neck module is provided with an EMA unit (Efficient Multi-Scale Attention Module with Cross-Spatial Learning); the head module is provided with a bounding box regression layer and a classification layer;

[0059] Step S4, training the tea pest identification model with the training set, testing the trained tea pest identification model with the test set, and verifying the tested tea pest identification model with the verification set; and obtaining a pest quantity prediction function by fitting the environmental data set;

[0060] Step S5: identifying tea pests using the verified tea pest identification model, and predicting the number of tea pests using the fitted pest number prediction function.

[0061] The step S1 is specifically as follows:

[0062] Collect a large number of tea images containing different types and numbers of pests and the environmental data corresponding to each of the tea images, annotate each of the tea images with pest boundary boxes, pest types, and pest numbers through Labelimg, perform brightness adjustment, contrast adjustment, or random rotation sample expansion operations on each of the annotated tea images, and then complete the preprocessing of each of the tea images, and construct a data set based on the preprocessed tea images;

[0063] Constructing an environmental data set based on the historical environmental data and the pest types and pest quantities of the corresponding tea leaves;

[0064] The historical environmental data includes at least relative humidity, rainfall probability, air pressure, snowfall, daily rainfall, monthly rainfall, wind direction, wind level, wind speed, sunrise time, sunset time, real-time temperature, perceived temperature, ultraviolet intensity, visibility, light radiation, sunshine duration, night duration, CO2 concentration, PM2.5 value and PM10 value.

[0065] The step S2 is specifically as follows:

[0066] The image dataset is divided into a training set, a test set, and a validation set based on a ratio of 6:2:2.

[0067] In step S3, the input module is used to input tea leaves images; the feature extraction module is used to extract pest features from tea leaves images; the neck module is used to fuse the extracted pest features; and the head module is used to output pest identification results.

[0068] In step S3, the formula of the WIoUv3 function is:

[0069] L WIoUv3 =rL WIoUv1 ;

[0070] r=β / δα β-δ ;

[0071]

[0072] L WIoUv1 =R WIoU L IoU ;

[0073]

[0074] Among them, L WIoUv3 represents the loss value of WIoUv3 function; r represents the non-monotonic focusing coefficient; L WIoUv1 represents the loss value of the WIoUv1 function; β represents the outlier coefficient; α and δ both represent hyperparameters that control the characteristics of the model; Indicates L WIoUv1 The monotonic focusing coefficient of represents the sliding average with momentum m; R WIoU ∈[1,e), represents a normal anchor box with enhanced quality; L IoU represents a high-quality anchor box whose quality has been reduced; (x, y) represents the coordinates of the upper left corner of the ordinary anchor box; (x gt ,y gt ) represents the coordinates of the upper left corner of the target box; W g Indicates the width of the pest boundary box; H g Indicates the height of the pest bounding box; represents the stable convergence factor.

[0075] The loss function of the traditional YOLOv8n network is the CIoU (Complete IoU) function, which mainly focuses on the loU between the predicted box and the real box, and cannot fully capture the relative position and size differences between the targets. It cannot adapt to the images of pests with mutilated limbs. In the face of low-quality data sources (tea images), it cannot accurately measure the performance of target detection. Therefore, the present invention uses the WIoUv3 function instead of the CIoU function as the loss function.

[0076] The core idea of ​​the WIoUv3 function is to use the outlier coefficient β instead of IoU to evaluate the quality of the anchor box. is dynamic, making L WIoUv3 It can make a real-time gradient allocation strategy; when the β value is small, the anchor box quality is high, and the loss function will assign a smaller gradient gain to it, so that the bounding box regression focuses on the anchor box of normal quality; when the β value is large, the anchor box quality is low, and the loss function will assign a larger gradient gain to it to avoid harmful gradients caused by low-quality anchor boxes, while not losing attention to the anchor boxes of normal quality. The WIoUv3 function can pay more attention to samples of normal quality, thereby improving the overall performance of the tea pest recognition model.

[0077] In the step S3, the SCConv unit is composed of an SRU subunit (Spatial Reconstruction Unit) and a CRU subunit (Channel Reconstruction Unit); the SRU subunit is connected to the CRU subunit;

[0078] The SRU subunit is used to suppress the spatial redundancy of the tea leaf image; and the CRU subunit is used to suppress the channel redundancy of the tea leaf image.

[0079] Since the traditional YOLOv8n network will generate more redundant information during feature extraction, in order to reduce the invalid calculation of the process and reduce the computational cost of feature extraction, the feature extraction module is improved through the SCConv unit.

[0080] The core step of the SRU subunit is separation-reconstruction to suppress the spatial redundancy of the feature map (tea image).

[0081] The separation of SRU subunits uses group normalization (GN) to evaluate the information content of the feature map and obtain the feature X out :

[0082]

[0083] Among them, GN() represents the group normalization function; X represents the input feature map, that is, the tea image; γ and β' both represent trainable affine transformation coefficients; x represents the feature in the feature map; μ represents the mean of the feature map; σ represents the standard deviation of the feature map; ε is a positive constant used to stabilize division;

[0084] Initialize γ randomly and normalize it to get the channel weight set W γ :

[0085]

[0086] Among them, w i represents the i-th channel weight; γ i represents the i-th γ; γ j represents the jth γ; C represents the total number of channels;

[0087] W γ With X out Multiply them, map the product to (0, 1) through the Sigmoid function, and use the threshold gating (Gate() function) to remove the W that is greater than the threshold. γ Set as a high information expression feature, weight W1, less than the threshold W γ Set as useless feature, weight is W2:

[0088] W n = Gate(Sigmoid(W γ *X out )), (n=1,2);

[0089] The reconstruction of SRU subunits is to transform the information-rich features into Features with less information Add and merge, and strengthen the information flow between the two to generate richer information features and suppress redundant features, and obtain a feature map X with rich spatial information w :

[0090]

[0091] in, Represents element-wise multiplication; It means element-by-element addition; The first part represents the reconstruction of rich information features; The second part represents the reconstruction of rich information features; Represents the first part of the reconstruction of less informative features; represents the second part of the reconstruction of less informative features; X w1 Indicates features with large amount of information after cross reconstruction; X w2 Indicates features with less information after cross-reconstruction.

[0092] The core steps of the CRU subunit are segmentation-conversion-fusion to reduce the channel redundancy of the feature map.

[0093] The segmentation of CRU subunits is to divide the feature map X w Split into two channel dimensions, αC and (1-α)C, and further use 1×1 convolution to compress the number of feature map channels to reduce the computational cost. w Divide into a channel-rich feature extractor X by splitting and compressing up and shallow channel feature extractor Xlow .

[0094] Conversion of CRU subunits, X up The efficient convolution GWC (Group-wise Convolution) and PWC (Point-wise Convolution) are combined to replace the standard K×K convolution to extract representative feature information while reducing the amount of calculation, and then the output is added and merged into a rich feature map Y1; X low Use 1×1 PWC to generate a complementary feature map, concatenate the generated features and reused features along the channel, and output the shallow hidden feature map Y2:

[0095]

[0096]

[0097] Among them, M G , All represent learnable weight matrices; + represents pixel-by-pixel addition operation; ∪ represents channel concatenation operation;

[0098] The fusion of CRU sub-units uses the Pooling operation to obtain the channel descriptor S with global spatial feature information. m , then stack the descriptor S1 of the upper channel and the descriptor S2 of the lower channel, and use the channel soft attention operation to generate important feature vectors β1 and β2, and finally fuse Y1 and Y2 to obtain the feature map Y with rich channel information:

[0099]

[0100] Y=β1Y1+β2Y2;

[0101] Among them, Pooling() represents the pooling function; H represents the height of the feature map space dimension; W represents the height of the feature map space dimension; Y m Represents the input feature map before the pooling operation; Y C Represents the input feature map in a specific channel; e represents a natural constant.

[0102] In step S3, the EMA unit is composed of a 1×1 convolution subunit, a 3×3 convolution subunit and a cross-space learning subunit.

[0103] In order to solve the problem that as the network depth increases, redundant information increases, resulting in the loss of important information and reduced recognition accuracy, an EMA unit is added to the neck module to improve the portability of the model while retaining a higher attention weight.

[0104] First, in order to learn different feature semantics, the input feature map X is divided into G sub-features along the channel direction X = [X0, X i ,...,X G-1 ]; In the 1×1 convolution subunit, the channel is encoded using the global adaptive average pooling operation along the X and Y directions, the encoded feature maps are connected, and the 1×1 convolution is shared; the output after the convolution operation is decomposed into two solution vectors, which are fitted using the activation function Sigmoid, and then the Re-weight adaptive feature variable selection is performed and the 1×1 convolution subunit is output.

[0105] The 3×3 convolution subunit uses a convolution kernel of size 3×3 to extract multi-scale features as output.

[0106] In the cross-spatial learning subunit, the output of the 1×1 convolution subunit is globally average pooled, activated by the global information encoding Softmax function, and then multiplied and dot-product aggregated with the output of the 3×3 convolution subunit to achieve cross-channel information interaction; at the same time, the output of the 3×3 convolution subunit is activated by the global information encoding Softmax function, and multiplied and aggregated with the output of the 1×1 convolution subunit after grouping and standardization to retain accurate spatial and positional features; the output results of the two matrix multiplications are activated by the Sigmoid function, and the global information is retained through the Re-weight adaptive feature variable to achieve pixel-level attention of different scales of the global context.

[0107] In step S3, the bounding box regression layer is used to predict the location of the pests and output a predicted pest bounding box;

[0108] The classification layer is used for prediction of pest types.

[0109] The step S4 is specifically as follows:

[0110] The tea pest identification model is trained by the training set, and the hyperparameters of the tea pest identification model including at least a random deactivation rate, a weight decay rate, and a learning rate are continuously optimized during the training process until the loss value of the loss function is less than a preset loss threshold;

[0111] The trained tea pest identification model is tested by the test set to determine whether the identification accuracy is greater than a preset accuracy threshold. If not, the training set is expanded to continue training; if yes, then:

[0112] The accuracy, recall rate, F1 score, average precision and mean average precision of the tea insect pest recognition model after the test are calculated by the verification set, and the performance of the tea insect pest recognition model is verified by the accuracy, recall rate, F1 score, average precision and mean average precision;

[0113] The number of pests in the environmental data set is taken as the dependent variable, the pest type and historical environmental data in the environmental data set are taken as independent variables, each of the dependent variables and independent variables is fitted to obtain the weight coefficient corresponding to each of the independent variables, and then the pest number prediction function is obtained.

[0114] The step S5 is specifically as follows:

[0115] The tea image to be identified and the real-time environmental data corresponding to the tea image to be identified are obtained. After the tea image to be identified is preprocessed for image enhancement, the tea image to be identified is input into a verified tea pest identification model, and a tea pest identification result containing at least a pest boundary box, pest type, and identification time is output; the real-time environmental data is input into a pest quantity prediction function, and a pest quantity prediction curve is output.

[0116] In summary, the advantages of the present invention are:

[0117] 1. By collecting a large number of tea images containing different types and numbers of pests, each tea image is preprocessed to construct a data set, and the data set is divided into a training set, a test set and a validation set; then a tea pest recognition model is created based on the input module, feature extraction module, neck module and head module of the YOLOv8n network, and the loss function of the tea pest recognition model is set to the WIoUv3 function; the feature extraction module is composed of SCConv units; the neck module is provided with an EMA unit; the head module is provided with a bounding box regression layer and a classification layer; then the tea pest recognition model is trained with the training set, the trained tea pest recognition model is tested with the test set, the tested tea pest recognition model is verified with the validation set, and finally the tea pest recognition model is used to recognize tea pests; that is, the traditional CIoU function is replaced by the WIoUv3 function, and a non-monotonic focusing system is introduced. The number is used to more accurately describe the boundary difference between the predicted box (predicted pest bounding box) and the true box (real pest bounding box), improve the detection effect of small targets, optimize the training process and enhance the generalization ability of the tea pest recognition model; the network structure of the traditional feature extraction module is improved through the SCConv unit, the spatial information and channel information are enriched, and redundant features (spatial redundancy, channel redundancy) are suppressed to maintain high accuracy and reduce the number of parameters of the tea pest recognition model; by introducing the EMA unit in the neck module, different feature semantics are effectively learned, and multi-scale pixel-level attention of the global context is realized, so that the tea pest recognition model pays more attention to the target area (pests), and improves the recognition accuracy of the tea pest recognition model for small targets, multiple targets and interference targets, that is, the tea pest recognition model created by the improved YOLOv8n network is used for tea pest recognition, which ultimately greatly improves the accuracy of tea pest recognition.

[0118] 2. By collecting a large number of tea images and the historical environmental data corresponding to each tea image, an environmental data set is constructed based on the historical environmental data and the pest types and pest quantities marked in the corresponding tea images; the pest quantity in the environmental data set is used as the dependent variable, and the pest type and historical environmental data in the environmental data set are used as independent variables. Each dependent variable and independent variable is fitted to obtain the weight coefficient corresponding to each variable, and then a pest quantity prediction function is obtained. Subsequently, the pest quantity prediction function can be used to predict the pest quantity, that is, to predict the pest quantity based on environmental data.

[0119] 3. Tea pests are identified through the tea pest identification model, and the growth number of tea pests is predicted through the pest quantity prediction function. It integrates the multimodal information of tea images and environmental data, and can carry out targeted prevention and control of different pests in different environments, effectively improving the pest control effect.

[0120] 4. By performing sample expansion operations such as brightness adjustment, contrast adjustment or random rotation on the labeled tea images, sufficient sample size can effectively prevent overfitting of the tea pest recognition model training, thereby effectively improving the generalization ability of the tea pest recognition model.

[0121] 5. By continuously optimizing the hyperparameters including at least the random inactivation rate, weight decay rate and learning rate during the training process of the tea pest identification model, the recognition accuracy of the trained tea pest identification model is tested through the test set, and the accuracy, recall rate, F1 score, average precision and average precision mean of the tested tea pest identification model are calculated through the validation set to verify the performance of the tea pest identification model. That is, multi-dimensional optimization and verification are performed during the training process of the tea pest identification model, thereby greatly improving the accuracy of tea pest identification.

[0122] 6. After preprocessing the tea image for image enhancement, the image is input into the tea pest recognition model to identify tea pests, thereby further improving the accuracy of tea pest recognition.

[0123] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A method for identifying and predicting tea pests by integrating multimodal information, characterized in that: The steps include: Step S1, collecting a large number of tea images containing pests of different types and numbers, and historical environmental data corresponding to each of the tea images, constructing an image dataset after preprocessing each of the tea images, and constructing an environmental dataset based on each of the historical environmental data and the tea images; Step S2, dividing the image data set into a training set, a test set and a validation set; Step S3, creating a tea pest recognition model based on the input module, feature extraction module, neck module and head module of the YOLOv8n network, and setting the loss function of the tea pest recognition model to the WIoUv3 function; The feature extraction module is composed of SCConv units; the neck module is composed of EMA units; the head module is composed of bounding box regression layers and classification layers; Step S4, training the tea pest identification model with the training set, testing the trained tea pest identification model with the test set, and verifying the tested tea pest identification model with the verification set; and obtaining a pest quantity prediction function by fitting the environmental data set; Step S5: identifying tea pests using the verified tea pest identification model, and predicting the number of tea pests using the fitted pest number prediction function.

2. A method for identifying and predicting tea pests by integrating multimodal information as claimed in claim 1, characterized in that: The step S1 is specifically as follows: Collect a large number of tea images containing different types and numbers of pests and the environmental data corresponding to each of the tea images, label each of the tea images with pest boundary boxes, pest types, and pest numbers through Label img, perform brightness adjustment, contrast adjustment, or random rotation sample expansion operations on each of the labeled tea images, and then complete the preprocessing of each of the tea images, and construct a data set based on each of the preprocessed tea images; Constructing an environmental data set based on the historical environmental data and the pest types and pest quantities of the corresponding tea leaves; The historical environmental data includes at least relative humidity, rainfall probability, air pressure, snowfall, daily rainfall, monthly rainfall, wind direction, wind level, wind speed, sunrise time, sunset time, real-time temperature, perceived temperature, ultraviolet intensity, visibility, light radiation, sunshine duration, night duration, CO2 concentration, PM2.5 value and PM10 value.

3. The method for identifying and predicting tea pests by integrating multimodal information according to claim 1, characterized in that: The step S2 is specifically as follows: The image dataset is divided into a training set, a test set, and a validation set based on a ratio of 6:2:

2.

4. The method for identifying and predicting tea pests by integrating multimodal information according to claim 1, characterized in that: In step S3, the input module is used to input tea leaves images; the feature extraction module is used to extract pest features from tea leaves images; the neck module is used to fuse the extracted pest features; and the head module is used to output pest identification results.

5. The method for identifying and predicting tea pests by integrating multimodal information according to claim 1, characterized in that: In step S3, the formula of the WIoUv3 function is: L WIoUv3 =rL WIoUv1 ; r=β / δα β-δ ; L WIoUv1 =R WIoU L IoU ; Among them, L WIoUv3 represents the loss value of WIoUv3 function; r represents the non-monotonic focusing coefficient; L WIoUv1 represents the loss value of the WIoUv1 function; β represents the outlier coefficient; α and δ both represent hyperparameters that control the characteristics of the model; Indicates L WIoUv1 The monotonic focusing coefficient of represents the sliding average with momentum m; R WIoU ∈[1,e), represents a normal anchor box with enhanced quality; L IoU represents a high-quality anchor box whose quality has been reduced; (x, y) represents the coordinates of the upper left corner of the ordinary anchor box; (x gt ,y gt ) represents the coordinates of the upper left corner of the target box; W g Indicates the width of the pest boundary box; H g Indicates the height of the pest bounding box; represents the stable convergence factor.

6. The method for identifying and predicting tea pests by integrating multimodal information according to claim 1, characterized in that: In step S3, the SCConv unit is composed of an SRU subunit and a CRU subunit; the SRU subunit is connected to the CRU subunit; The SRU subunit is used to suppress the spatial redundancy of the tea leaf image; and the CRU subunit is used to suppress the channel redundancy of the tea leaf image.

7. The method for identifying and predicting tea pests by integrating multimodal information according to claim 1, characterized in that: In step S3, the EMA unit is composed of a 1×1 convolution subunit, a 3×3 convolution subunit and a cross-space learning subunit.

8. The method for identifying and predicting tea pests by integrating multimodal information according to claim 1, characterized in that: In step S3, the bounding box regression layer is used to predict the location of the pests and output a predicted pest bounding box; The classification layer is used for prediction of pest types.

9. The method for identifying and predicting tea pests by integrating multimodal information according to claim 1, characterized in that: The step S4 is specifically as follows: The tea pest identification model is trained by the training set, and the hyperparameters of the tea pest identification model including at least a random deactivation rate, a weight decay rate, and a learning rate are continuously optimized during the training process until the loss value of the loss function is less than a preset loss threshold; The trained tea pest identification model is tested by the test set to determine whether the identification accuracy is greater than a preset accuracy threshold. If not, the training set is expanded to continue training; if yes, then: Calculating the accuracy, recall rate, F1 score, average precision and mean of average precision of the tested tea pest identification model through the verification set, and verifying the performance of the tea pest identification model through the accuracy, recall rate, F1 score, average precision and mean of average precision; The number of pests in the environmental data set is taken as the dependent variable, the pest type and historical environmental data in the environmental data set are taken as independent variables, each of the dependent variables and independent variables is fitted to obtain the weight coefficient corresponding to each of the independent variables, and then the pest number prediction function is obtained.

10. The method for identifying and predicting tea pests by integrating multimodal information according to claim 1, characterized in that: The step S5 is specifically as follows: The tea image to be identified and the real-time environmental data corresponding to the tea image to be identified are obtained. After the tea image to be identified is preprocessed for image enhancement, the tea image to be identified is input into a verified tea pest identification model, and a tea pest identification result containing at least a pest boundary box, pest type, and identification time is output; the real-time environmental data is input into a pest quantity prediction function, and a pest quantity prediction curve is output.

Images