Bee pest detection method and system based on multi-modal data

Abstract

The invention provides a bee pest detection method and system based on multi-modal data, and relates to the technical field of bee protection.The method comprises the steps that a pest priori knowledge base of bees is constructed, picture data serves as input, and the probability of a first prediction pest category is recognized through an image classifier; converting the text data into a text vector, taking the text vector as input, and identifying the probability of a second prediction pest category by using the text classifier; converting the geographical location information and the time into numerical features, taking the numerical features as input, obtaining the probability of a third predicted pest category by using a pest category prediction model, and forming a feature matrix by the probability of the first predicted pest category, the probability of the second predicted pest category and the probability of the third predicted pest category, the regional pest prediction model is used for obtaining the pests of the bees in the to-be-detected region, the problem that in the prior art, time and labor are wasted when bee pest multi-mode reported data detection is carried out is solved, and the method is suitable for detecting the pests of the bees.

Description

Technical Field

[0001] The present invention relates to the technical field of bee protection, and particularly to a method and system for detecting pests of bees based on multi-modal data. Background Art

[0002] Bees are important pollinating insects and play a key role in agricultural production and ecological balance. Monitoring pests of bees can timely detect and prevent the spread of diseases, reduce the mortality rate of bees, avoid the loss of bee colonies and the decline in the output of bee products caused by large-scale outbreaks of pests, reduce economic losses, and thus ensure the sustainable development of the beekeeping industry.

[0003] Traditional methods for detecting pests of bees mainly rely on manual processing of the reported data of pests of bees in the area to be detected, and these data include pictures, texts, geographical locations, and time information. However, manually processing these data to detect pests of bees is not only time-consuming and laborious, but also has a low accuracy rate. Summary of the Invention

[0004] The technical problem to be solved by the present invention: The present invention provides a method and system for detecting pests of bees based on multi-modal data, and solves the problem of time-consuming and laborious detection of multi-modal reported data of pests of bees in the prior art.

[0005] The technical solution adopted by the present invention to solve the above technical problem: A method for detecting pests of bees based on multi-modal data, comprising the following steps:

[0006] S1. Construct a prior knowledge base of pests of bees, where the prior knowledge base includes picture data of pests, text description data of pests, pest categories, geographical locations where pests appear, and times when pests appear;

[0007] S2. Taking the picture data of pests as input and the pest category as output, train a residual neural network model to obtain a pest image classifier; convert the text description data of pests into text vectors, taking the text vectors as input and the pest category as output, train a Bert model to obtain a pest text classifier; convert the geographical location where pests appear and the time when pests appear into numerical features, taking the numerical features as input and the pest category as output, train a random forest model to obtain a pest category prediction model;

[0008] S3. Obtain picture data, text data, geographical location information, time, and true values of pest categories of pests of bees;

[0009] S4. Take the image data as input, and use an image classifier to identify the probability of the first predicted pest category; convert the text data into a text vector, take the text vector as input, and use a text classifier to identify the probability of the second predicted pest category; convert the geographical location information and time into numerical features, take the numerical features as input, and use a pest category prediction model to obtain the probability of the third predicted pest category;

[0010] S5. Combine the probability of the first predicted pest category, the probability of the second predicted pest category, and the probability of the third predicted pest category to form a feature matrix;

[0011] S6. Take the feature matrix as input and the true value of the pest category as output, train a support vector machine model to obtain a regional pest prediction model;

[0012] S7. Obtain the image data, text data, geographical location information, and time of the pests of bees in the area to be detected, use an image classifier, a text classifier, and a pest category prediction model to obtain a feature matrix, and then use the regional pest prediction model to obtain the pests of bees in the area to be detected.

[0013] Further, the residual neural network model is a ResNet34 model.

[0014] Further, the numerical features are: X = [G coord , G one-hot , T season , T month-sin , where X represents the numerical features, G coord represents the two-dimensional vector corresponding to the latitude and longitude coordinates, G one-hot represents the one-hot encoding of the geographical location where the pest appears, T season represents the one-hot encoding of the season, and T month represents the periodic encoding of the month.

[0015] Further, the one-hot encoding process of the geographical location where the pest appears includes: dividing the geographical location into multiple regions, representing them with 0s and arranging them in sequence, and replacing the 0 corresponding to the region where the pest appears with 1 to obtain the one-hot encoding of the geographical location where the pest appears.

[0016] Further, the one-hot encoding process of the season includes: representing the four seasons with 0s and arranging them in sequence, and replacing the 0 corresponding to the season where the time is located with 1 to obtain the one-hot encoding of the season.

[0017] Further, the periodic encoding formula of the month is: where m represents the month where the time is located.

[0018] The present invention also provides a pest detection system for bees based on multi-modal data, which implements the pest detection method for bees based on multi-modal data as described above. The system includes an image classifier, a text classifier, a pest category prediction model, and a regional pest prediction model. The image classifier is used to identify the probability of the first predicted pest category from the picture data of pests. The text classifier is used to identify the probability of the second predicted pest category based on the text data of pests. The pest category prediction model is used to obtain the probability of the third predicted pest category based on geographical location information and time. The regional pest prediction model is used to obtain the pests of bees in the area to be detected by integrating the probability of the first predicted pest category, the probability of the second predicted pest category, and the probability of the third predicted pest category.

[0019] Advantages of the present invention: The present invention provides a pest detection method and system for bees based on multi-modal data. By constructing a prior knowledge base of pests of bees, using picture data as input, and using an image classifier to identify the probability of the first predicted pest category; converting text data into text vectors, using the text vectors as input, and using a text classifier to identify the probability of the second predicted pest category; converting geographical location information and time into numerical features, using the numerical features as input, and using a pest category prediction model to obtain the probability of the third predicted pest category, and forming a feature matrix with the probability of the first predicted pest category, the probability of the second predicted pest category, and the probability of the third predicted pest category, and using a regional pest prediction model to obtain the pests of bees in the area to be detected, which solves the problem of time-consuming and laborious detection of multi-modal reported data of bee pests in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a schematic flow chart of a pest detection method for bees based on multi-modal data provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] Aiming at the problem of time-consuming and laborious pest detection of bees in the prior art, the present invention provides a pest detection method for bees based on multi-modal data, as Figure 1 shown, including the following steps:

[0022] S1. Construct a prior knowledge base of pests of bees, where the prior knowledge base includes picture data of pests, text description data of pests, pest categories, geographical locations where pests appear, and time when pests appear;

[0023] Specifically, bee pest monitoring involves a large amount of data types and data volumes, including pest pictures, types and quantities of text descriptions, distribution areas, and occurrence times, etc. Therefore, relevant pest data is obtained through field investigations, historical documents, etc. to construct a prior knowledge base.

[0024] S2. Use the pest picture data as the input and the pest category as the output to train a residual neural network model to obtain a pest image classifier; convert the text description data of the pest into a text vector, use the text vector as the input and the pest category as the output to train a Bert model to obtain a pest text classifier; convert the geographical location where the pest appears and the time when the pest appears into numerical features, use the numerical features as the input and the pest category as the output to train a random forest model to obtain a pest category prediction model;

[0025] Specifically, the residual neural network model is a ResNet34 model. Through S2, a pest image classifier capable of identifying pests in the pest picture data, a pest text classifier capable of identifying pests in the text description data of the pest, and a pest category prediction model capable of inferring the pest category that appears at a geographical location and time are obtained. The numerical features are: X =

[0026] [Gcoord, Gone-hot, Tseason, Tmonth-sin], where X represents the numerical features, Gcoord represents the two-dimensional vector corresponding to the longitude and latitude coordinates, Gone-hot represents the one-hot encoding of the geographical location where the pest appears, Tseason represents the one-hot encoding of the season, and Tmonth represents the periodic encoding of the month. The one-hot encoding process of the geographical location where the pest appears includes: dividing the geographical location into multiple regions and representing them with 0s, arranging them in sequence, and replacing the 0 corresponding to the region where the pest appears with 1 to obtain the one-hot encoding of the geographical location where the pest appears. The one-hot encoding process of the season includes: representing the four seasons with 0s, arranging them in sequence, and replacing the 0 corresponding to the season where the time is located with 1 to obtain the one-hot encoding of the season. The periodic encoding formula of the month is: where m represents the month when the time is located.

[0027] S3. Obtain the picture data, text data, geographical location information, time, and true pest category value of the pests of bees;

[0028] Specifically, a data set is obtained therefrom as the training data for the subsequent regional pest prediction model.

[0029] S4. Use the picture data as the input and use the image classifier to identify the probability of the first predicted pest category; convert the text data into a text vector, use the text vector as the input and use the text classifier to identify the probability of the second predicted pest category; convert the geographical location information and time into numerical features, use the numerical features as the input and use the pest category prediction model to obtain the probability of the third predicted pest category;

[0030] Specifically, three results predicted by three different data sources are obtained through S4.

[0031] S5. Combine the probabilities of the first predicted pest category, the second predicted pest category, and the third predicted pest category to form a feature matrix;

[0032] S6. Use the feature matrix as the input and the true value of the pest category as the output to train a support vector machine model to obtain a regional pest prediction model;

[0033] Specifically, the most accurate prediction result is selected from the above three prediction results through the regional pest prediction model.

[0034] S7. Obtain the picture data, text data, geographical location information, and time of the pests of bees in the area to be detected, use the image classifier, text classifier, and pest category prediction model to obtain a feature matrix, and then use the regional pest prediction model to obtain the pests of bees in the area to be detected.

[0035] Specifically, use the image classifier, text classifier, and pest category prediction model to obtain a feature matrix. The implementation method includes S4, S5, and inputting the feature matrix into the regional pest prediction model. In this way, the pests of bees in the area to be detected can be accurately obtained.

[0036] The present invention also provides a pest detection system for bees based on multi-modal data, which implements the pest detection method for bees based on multi-modal data as described above. The system includes an image classifier, a text classifier, a pest category prediction model, and a regional pest prediction model. The image classifier is used to identify the probability of the first predicted pest category from the picture data of the pests. The text classifier is used to identify the probability of the second predicted pest category based on the text data of the pests. The pest category prediction model is used to obtain the probability of the third predicted pest category based on the geographical location information and time. The regional pest prediction model is used to comprehensively obtain the pests of bees in the area to be detected based on the probabilities of the first predicted pest category, the second predicted pest category, and the third predicted pest category.

Claims

1. A method for detecting pests of bees based on multimodal data, characterized in that: The following steps are involved: S1. Construct a priori knowledge base of pests for bees, wherein the priori knowledge base includes image data of pests, text description data of pests, pest categories, geographical locations where pests appear, and time when pests appear; S2. Take the image data of pests as input and the pest category as output, train the residual neural network model, and obtain a pest image classifier; convert the text description data of pests into text vectors, take the text vectors as input and the pest category as output, train the Bert model, and obtain a pest text classifier; convert the geographical location and time of pest appearance into numerical features, take the numerical features as input and the pest category as output, train the random forest model, and obtain a pest category prediction model; S3, obtaining the image data, text data, geographic location information, time and true value of the pest category of the bees; S4, using the image data as input, identifying the probability of a first predicted pest category using an image classifier; The text data is converted into a text vector, and the text vector is used as input to identify the probability of the second predicted pest category by using a text classifier; the geographic location information and time are converted into numerical features, and the numerical features are used as input to obtain the probability of the third predicted pest category by using a pest category prediction model; S5, forming a feature matrix by combining the probability of the first predicted pest category, the probability of the second predicted pest category and the probability of the third predicted pest category; S6, taking the feature matrix as input and the true value of the pest category as output, training the support vector machine model to obtain a regional pest prediction model; S7. Obtain image data, text data, geographic location information and time of bee pests in the area to be detected, use image classifier, text classifier and pest category prediction model to obtain feature matrix, and then use regional pest prediction model to obtain bee pests in the area to be detected.

2. The bee pest detection method based on multimodal data according to claim 1, characterized in that: The residual neural network model is a ResNet34 model.

3. The bee pest detection method based on multimodal data according to claim 1, characterized in that: The numerical feature is: X = [G coord ,G one-hot ,T season ,T month-sin ], where X represents the numerical feature, G coord Represents the two-dimensional vector corresponding to the longitude and latitude coordinates, G one-hot The one-hot encoding of the geographic location where the pest appears, T season Indicates the one-hot encoding of the season, T month A recurring code representing the month.

4. The bee pest detection method based on multimodal data according to claim 3, characterized in that: The one-hot encoding process of the geographical location where the pests appear includes: dividing the geographical location into multiple areas, representing them with 0, arranging them in sequence, replacing the 0 corresponding to the area where the geographical location where the pests appear is located with 1, and obtaining the one-hot encoding of the geographical location where the pests appear.

5. The bee pest detection method based on multimodal data according to claim 3, characterized in that: The one-hot encoding process of the season includes: representing the four seasons with 0, arranging them in sequence, replacing the 0 corresponding to the season of the time with 1, and obtaining the one-hot encoding of the season.

6. The bee pest detection method based on multimodal data according to claim 3, characterized in that: The periodic coding formula for the month is: Among them, m represents the month.

7. A bee pest detection system based on multimodal data, characterized in that: The method for detecting bees' pests based on multimodal data as described in claim 1 is implemented, wherein the system includes an image classifier, a text classifier, a pest category prediction model, and a regional pest prediction model, wherein the image classifier is used to identify the probability of a first predicted pest category based on the image data of the pests, the text classifier is used to identify the probability of a second predicted pest category based on the text data of the pests, the pest category prediction model is used to obtain the probability of a third predicted pest category based on geographic location information and time, and the regional pest prediction model is used to obtain the pests of the bees in the area to be detected by combining the probabilities of the first predicted pest category, the second predicted pest category, and the third predicted pest category.

Images