Vineyard pest and disease fine-grained recognition method based on attribute feature knowledge graph

Abstract

The present application relates to a grape garden pest and disease fine-grained recognition method based on attribute feature knowledge graph, comprising: making a grape garden pest and disease dataset; constructing a grape garden pest and disease attribute feature knowledge graph; extracting color features, global texture features and contour features of the input pest image to obtain a traditional feature vector through splicing operation; extracting input training pest image features and test pest image features through a GPKG-ViT network; inputting the training pest image features or test pest image features into a classification layer to obtain a predicted pest category. The present application has the beneficial effect that the present application introduces a knowledge graph into a deep learning network, realizes fine-grained recognition of grape garden pests through the introduction of fine-grained attribute features and pest entity association feature information, and is a highly effective and highly universal method.

Description

Technical Field

[0001] This invention relates to the field of smart agriculture technology, and more specifically, to a fine-grained identification method for vineyard pests and diseases based on attribute feature knowledge graphs. Background Technology

[0002] Pests and diseases are a major cause of reduced crop yields and quality in vineyards. Utilizing computer technology to automatically identify pests and diseases, and assisting domain experts in developing scientific prevention and control strategies, is an important way to improve vineyard production. Deep learning, as a key technology in this field, overcomes the shortcomings of traditional image classification methods, such as poor feature extraction capabilities and low efficiency, and is widely used in the identification and diagnosis of crop pests and diseases. However, due to the large variety of crop pests and diseases, their complex morphological attributes, and poor deep-level relationships between entities, the application results are often not ideal. This invention, guided by domain experts, constructs a pest attribute feature knowledge graph and proposes a fine-grained vineyard pest and disease identification method based on this graph. By introducing the knowledge graph into a deep learning network, accurate identification of vineyard pests and diseases is achieved. This method can serve as a knowledge base for downstream applications such as vineyard pest and disease information retrieval, intelligent question answering, and intelligent recommendation, and can be effectively applied to agricultural production aspects such as crop variety selection and pest and disease control. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the prior art and to provide a fine-grained identification method for vineyard pests and diseases based on attribute feature knowledge graphs.

[0004] Firstly, a fine-grained identification method for vineyard pests and diseases based on attribute feature knowledge graphs is provided, including:

[0005] Step 1: Create a dataset of vineyard pests and diseases;

[0006] Step 2: Construct a knowledge graph (GPKG) of the attributes and characteristics of vineyard diseases and pests;

[0007] Step 3: Extract color features f from the input pest image. c Global texture features f t and contour features f o The traditional feature vector f is obtained through concatenation. MF ;

[0008] Step 4: Extract features f from the input training insect pest images using the GPKG-ViT network. train and test pest image features f test ;

[0009] Step 5: Train the pest image features f train and test pest image features f testInput the classification layer to obtain the predicted pest category.

[0010] Preferably, step 1 includes:

[0011] Step 1.1: Obtain the IP102 dataset and construct the GP21 dataset based on the IP102 dataset;

[0012] Step 1.2: Divide the GP21 dataset into the GP21 test set and the GP21 training set according to the original proportions of the IP102 dataset.

[0013] Preferably, step 2 includes:

[0014] Step 2.1: Use the Scrapy framework to crawl vineyard pest data;

[0015] Step 2.2: Transform the vineyard pest data into a standardized vineyard pest and disease knowledge corpus using regular expressions; for semi-structured data, directly extract entities; for unstructured data, use the deep learning model Bi-LSTM-CRF for entity extraction.

[0016] Step 2.3: Based on the vineyard pest and disease knowledge corpus, construct a vineyard pest and disease attribute feature knowledge graph (GPKG); the instance set is defined as a triple of <pest category, relationship, attribute feature>, and the graph database Neo4j is selected as the knowledge storage method.

[0017] Preferably, step 3 includes:

[0018] Step 3.1: Calculate the color moments of the input pest image, which will be used as the color features f of the input pest image. c ;

[0019] Step 3.2: Extract image texture features using local binary mode and gray-level co-occurrence matrix respectively, and then concatenate them. The resulting vector is used as the global texture feature f. t ;

[0020] Step 3.3: Extract contour features f based on the Canney edge detection algorithm o ;

[0021] Step 3.4: Obtain the traditional feature vector f through concatenation operation. MF :

[0022] f MF =Concat(f c ,f t ,f o )

[0023] Concat represents the concatenation operation.

[0024] Preferably, step 4 includes:

[0025] Step 4.1, During training: Based on the labels of the input pest images, index the knowledge graph to obtain the attribute feature vector f of the corresponding node in the knowledge graph for that type of pest. CF Then, compared with the traditional feature vector f MF Perform cosine similarity calculation to obtain the similarity loss.

[0026]

[0027] Where n represents the dimension of the feature vector, which is equal to the total number of pest categories;

[0028] During testing: Utilize the traditional feature vector f for each image. MF The cosine similarity is calculated between the feature vectors and the feature vectors corresponding to all nodes representing pest categories in GPKG, and the combined feature vectors are used to obtain the attribute similarity feature vector. Use l k The node index represents the pest category. Represented as:

[0029]

[0030] Step 4.2: Extract high-level semantic features of the image using ViT, and use the ViT head output as the final representation feature vector f. SF ;

[0031] Step 4.3: Combine image attribute features extracted based on knowledge graphs and high-level semantic representation features extracted based on ViT for classification training, and output the training pest image features f. train and test pest image features f test :

[0032] f train =f CF +f SF

[0033]

[0034] Preferably, in step 5, the training pest image features f train Or test the characteristics of pest images f test Input to the classification layer to obtain the predicted pest category; model loss Using cross-entropy loss function Sum and cosine loss function express:

[0035]

[0036]

[0037] In the above formula, y i and These represent the true and predicted labels of the input pest image, respectively. express The predicted probability.

[0038] In a second aspect, a computer storage medium is provided, characterized in that the computer storage medium stores a computer program; when the computer program is run on a computer, it causes the computer to execute any of the fine-grained identification methods for vineyard pests and diseases described in the first aspect.

[0039] Thirdly, a computer program product is provided, characterized in that, when the computer program product is run on a computer, it causes the computer to execute any of the fine-grained identification methods for vineyard pests and diseases described in the first aspect.

[0040] The beneficial effects of this invention are:

[0041] (1) This invention introduces knowledge graphs into deep learning networks and achieves fine-grained identification of vineyard pests by introducing fine-grained attribute features and pest entity association features.

[0042] (2) This invention performs sample analysis at the object level, combining the pest attribute feature knowledge graph as a branch with another deep learning module branch, which is a highly effective and universal method.

[0043] (3) This invention can serve as the knowledge base for downstream applications such as vineyard pest and disease information retrieval, intelligent question answering, and intelligent recommendation. It can be effectively applied to agricultural production aspects such as crop variety selection and pest and disease control. Attached Figure Description

[0044] Figure 1 The flowchart shows a fine-grained identification method for vineyard pests and diseases based on attribute feature knowledge graphs.

[0045] Figure 2 A schematic diagram of the GPKG knowledge graph for the attribute characteristics of vineyard pests;

[0046] Figure 3 Diagram of the GPKG-ViT network model;

[0047] Figure 4 A visual comparison chart showing the pest identification performance of GPKG-ViT and ViT. Detailed Implementation

[0048] The present invention will be further described below with reference to embodiments. The description of the embodiments below is only for the purpose of helping to understand the present invention. It should be noted that those skilled in the art can make several modifications to the present invention without departing from the principle of the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

[0049] Example 1

[0050] A knowledge graph is a knowledge base that integrates data using graph-based data models or topological structures. It can accurately describe complex knowledge within a domain and is widely used in fields such as intelligent search and personalized recommendations. In the application of knowledge graphs to the problem of crop pest and disease identification, it overcomes the shortcomings of traditional deep learning techniques, such as poor classification of multiple pest categories, difficulty in identifying complex pest morphological attributes, and poor deep-level relationships between pest entities, achieving ideal application results.

[0051] This invention addresses the problems of numerous pests in vineyards, including their diverse types, complex morphologies, and poor correlations. It fully leverages the advantages of knowledge graphs in describing the attribute features of pest entities and the relationships between them, proposing a fine-grained vineyard pest identification model, GPKG-ViT, based on attribute feature knowledge graphs. ViT (Visual Transformer) is used as the backbone network for extracting high-level image representations, and the fine-grained attribute features and pest entity correlation features provided by the knowledge graph are combined for vineyard pest classification research.

[0052] Specifically, such as Figure 1 As shown, the fine-grained identification method for vineyard pests and diseases based on attribute feature knowledge graphs provided by this invention includes:

[0053] Step 1: Create a dataset of vineyard pests and diseases.

[0054] Step 1 can be referred to as the vineyard pest image dataset acquisition and preprocessing stage, which includes the following steps:

[0055] Step 1.1: Obtain the IP102 dataset and construct the GP21 dataset based on it. For example, the IP102 dataset includes 75,222 samples collected from professional agricultural and entomological websites. Under the guidance of agricultural experts, 21 common vineyard pests were selected from these samples to form the GP21 dataset.

[0056] Step 1.2: Divide the GP21 dataset into the GP21 test set and the GP21 training set according to the original proportions of the IP102 dataset. For example, after the division, the total number of training samples and test samples will be 10303 and 1714, respectively.

[0057] Step 2: Construct a knowledge graph (GPKG) of the attributes and characteristics of vineyard diseases and pests.

[0058] Step 2 comprises two stages: the construction of a knowledge graph of pest attributes and the transformation of the knowledge graph. Step 2 includes:

[0059] Step 2.1: Use the Scrapy framework to crawl vineyard pest data. For example, using the Scrapy framework and knowledge bases such as professional agricultural websites, insect science websites, Wikipedia, and Baidu Encyclopedia, a total of 1264 data entries were crawled, including data on 21 common vineyard pests and diseases such as the green mirid bug, the large green leafhopper, and the grape two-spotted leafhopper.

[0060] Step 2.2: Transform vineyard pest data into a standardized vineyard pest and disease knowledge corpus using regular expressions; for semi-structured data, directly extract entities; for unstructured data, use the deep learning model Bi-LSTM-CRF for entity extraction.

[0061] Step 2.3: Based on the corpus of knowledge about vineyard pests and diseases, construct a knowledge graph GPKG of attributes and characteristics of vineyard pests and diseases; the instance set is defined as a triple of <pest category, relationship, attribute characteristics>, and the graph database Neo4j is selected as the knowledge storage method.

[0062] like Figure 2 As shown, the knowledge graph GPKG is mapped to a trainable deep learning network using the GAT network; the nodes of the knowledge graph include pest categories N. l Pest characteristics N f Two types:

[0063]

[0064]

[0065] In the formula, n and m represent the total number of pest categories and the number of all attribute nodes in the graph, respectively.

[0066] Step 3: Extract color features f from the input pest image. c Global texture features f t and contour features f o The traditional feature vector f is obtained through concatenation. MF .

[0067] Step 3 is the manual feature extraction stage, which includes the following steps:

[0068] Step 3.1: Calculate the color moments of the input pest image, which will be used as the color features f of the input pest image. c .

[0069] Step 3.2: Extract image texture features using Local Binary Patterns and Gray-level Co-occurrence Matrix, respectively, and concatenate them. The resulting vector is used as the global texture feature f. t .

[0070] Step 3.3: Extract contour features f based on the Canney edge detection algorithm o .

[0071] Step 3.4: Obtain the traditional feature vector f through concatenation operation. MF :

[0072] f MF =Concat(f c ,f t ,f o )

[0073] Here, Concat represents the concatenation operation, and the above-mentioned traditional feature vector can also be called a handmade feature vector.

[0074] Step 4: Extract features f from the input training insect pest images using the GPKG-ViT network. train and test pest image features f test .

[0075] like Figure 3 As shown, step 4 is the feature acquisition stage, which includes the following steps:

[0076] Step 4.1, During training: Based on the labels of the input pest images, index the knowledge graph to obtain the attribute feature vector f of the corresponding node in the knowledge graph for that type of pest. CF Then, compared with the traditional feature vector f MF Perform cosine similarity calculation to obtain the similarity loss.

[0077]

[0078] Where n represents the dimension of the feature vector, which is equal to the total number of pest categories;

[0079] During testing: Utilize the traditional feature vector f for each image. MF The cosine similarity is calculated between the feature vectors and the feature vectors corresponding to all nodes representing pest categories in GPKG, and the combined feature vectors are used to obtain the attribute similarity feature vector. Use l k The node index represents the pest category. Represented as:

[0080]

[0081] Step 4.2: Extract high-level semantic features of the image using ViT, and use the ViT head output as the final representation feature vector f. SF ;

[0082] Step 4.3: Combine image attribute features extracted based on knowledge graphs and high-level semantic representation features extracted based on ViT for classification training, and output the training pest image features f. train and test pest image features f test :

[0083] f train =f CF +f SF

[0084]

[0085] Step 5: Train the pest image features f train and test pest image features f test Input the classification layer to obtain the predicted pest category.

[0086] Step 5 is the prediction and classification stage, in which the training pest image features f are used. train Or test the characteristics of pest images f test Input to the classification layer to obtain the predicted pest category; model loss Using cross-entropy loss function Sum and cosine loss function express:

[0087]

[0088]

[0089] In the above formula, y i and These represent the true and predicted labels of the input pest image, respectively. express The predicted probability.

[0090] Example 2

[0091] To verify the results, this embodiment used the GP21 dataset of vineyard pests, which includes 21 common vineyard pests, as shown in Table 1. The total number of training and testing samples in GP21 is 10303 and 1714, respectively.

[0092] Table 1. Pest Categories in the GP21 Dataset

[0093] Green-spotted bug Large Green Leafhopper Grape two-spot leafhopper Grape phylloxera wheat aphid Kang-style mealybug Whiteflies Spotted Lanternfly Spodoptera litura Grape clearwing moth Grape hawk moth Stinging moth Grape gall mite Grape short-haired mite Lateral tarsiforme Flea Ten-star firefly leaf beetle Grape Tiger Longhorn Beetle Wireworm thrips Red Spider

[0094] Two comparison schemes were designed for the experiment:

[0095] Option 1 is based on the GP21 dataset and compared with different methods to verify the basic classification accuracy of the overall model. Because the fine-grained pest identification method in this embodiment relies on the ViT model, the overall model should show a more ideal performance than ViT. The results are shown in Table 2 below:

[0096] Table 2 Performance Comparison of Different Models

[0097] pre-trained model Accuracy F1 Precision Recall VGG-16 85.13 77.08 79.34 75.45 ResNet-152 87.49 79.59 81.07 79.03 Inception-V3 87.18 79.11 80.71 78.48 Xception 84.49 75.78 77.95 74.58 MobileNet 85.19 76.84 78.70 75.59 SqueezeNet 76.79 67.28 70.89 65.23 ViT 89.57 83.05 84.98 81.70 GPKG-ViT 91.21 85.95 87.52 84.99

[0098] Table 2 lists the performance of pre-trained networks VGG-16, ResNet-152, Inception-V3, Xception, MobileNet, SqueezeNet, and ViT on the GP21 test set. As shown in Table 2, the ViT model significantly outperforms the other models in both accuracy and F1 score. Compared to ResNet-152, one of the most frequently used models in current visual tasks, ViT improves accuracy and F1 scores by 2.08% and 3.46%, respectively. This indicates that the high-level representations extracted by ViT can more finely integrate global and local information in insect pest images. Therefore, it is reasonable to use ViT as the backbone network to construct the GPKG-ViT model in this study.

[0099] The performance of GPKG-ViT is shown in the last row of Table 2. Compared to ViT, GPKG-ViT's accuracy and F1 score are improved by 1.64% and 2.90%, respectively. This is because ViT is insufficient in recognizing objects with similar shapes, while the knowledge graph can provide detailed information about different types of pests, thus assisting ViT in distinguishing pest types. For a visual comparison of the pest recognition performance of GPKG-ViT and ViT, please refer to [link to relevant documentation]. Figure 4 .

[0100] Option 2 is the ablation experiment. To further analyze the effect of knowledge graphs on improving the classification performance of vineyard pests and diseases, we conducted three sets of ablation tests, and the results are shown in Table 3:

[0101] Table 3 Ablation Test Results

[0102]

[0103] Note: In the table, "w / o" indicates a removal operation. MF represents handcrafted features, and KG represents a knowledge graph.

[0104] As shown in the table, removing the branch containing the knowledge graph (w / o MF∪KG) reduces the model's accuracy and F1 score by 1.64% and 2.90%, respectively. Removing handcrafted features (w / o MF) and removing the knowledge graph (w / o KG) reduces the model's F1 score by 1.35% and 1.55%, respectively, and the accuracy by 2.32% and 2.36%, respectively. These results indicate that: 1) Introducing a knowledge graph is effective in helping ViT obtain more accurate pest information; 2) Using only traditional features and knowledge graphs has little effect on improving model performance, mainly because traditional feature extraction methods have deficiencies in expressing high-level semantic information of images, and using only knowledge graphs cannot effectively train graph convolutional networks, resulting in insufficient representation of node feature vectors.

[0105] The experimental results show that the present invention has achieved relatively ideal results in the identification of pests in vineyards.

Claims

1. A fine-grained identification method for vineyard pests and diseases based on attribute feature knowledge graphs, characterized in that, include: Step 1: Create a dataset of images of vineyard pests and diseases; Step 2: Construct a knowledge graph (GPKG) of the attributes and characteristics of vineyard diseases and pests; Step 3: Extract color features f from the input pest image. c Global texture features f t and contour features f o The traditional feature vector f is obtained through concatenation. MF ; Step 3 includes: Step 3.1: Calculate the color moments of the input pest image, which will be used as the color features f of the input pest image. c ; Step 3.2: Extract image texture features using local binary mode and gray-level co-occurrence matrix respectively, and then concatenate them. The resulting vector is used as the global texture feature f. t ; Step 3.3: Extract contour features f based on the Canney edge detection algorithm o ; Step 3.4: Obtain the traditional feature vector f through concatenation operation. MF : f MF =Concat(f c ,f t ,f o ) Wherein, Concat represents the concatenation operation; Step 4: Extract features f from the input training insect pest images using the GPKG-ViT network. train and test pest image features f test ; Step 4 includes: Step 4.1, During training: Based on the labels of the input pest images, index them in the knowledge graph to obtain the attribute feature vector f of the corresponding node of the input pest image in the knowledge graph. CF Then, compared with the traditional feature vector f MF Perform cosine similarity calculation to obtain the similarity loss. Where n represents the dimension of the feature vector, which is equal to the total number of pest categories; During testing: The traditional feature vector f of each input pest image is used. MF The cosine similarity is calculated between the feature vectors and the feature vectors corresponding to all nodes representing pest categories in GPKG, and the combined feature vectors are used to obtain the attribute similarity feature vector. Use l k The node index represents the pest category. Represented as: Step 4.2: Use ViT to extract high-level semantic features from the input pest image, and use the ViT head output as the final representation feature vector f. SF ; Step 4.3: Combine the attribute feature vector f CF Attribute similarity feature vector and the final representation feature vector f SF Used for classification training, outputting training pest image features f. train and test pest image features f test : f train =f CF +f SF Step 5: Train the pest image features f train Or test the characteristics of pest images f test Input the classification layer to obtain the predicted pest category.

2. The method for fine-grained identification of vineyard pests and diseases based on attribute feature knowledge graphs according to claim 1, characterized in that, Step 1 includes: Step 1.1: Obtain the IP102 dataset and construct the GP21 dataset based on the IP102 dataset; Step 1.2: Divide the GP21 dataset into the GP21 test set and the GP21 training set according to the original proportions of the IP102 dataset.

3. The method for fine-grained identification of vineyard pests and diseases based on attribute feature knowledge graphs according to claim 2, characterized in that, Step 2 includes: Step 2.1: Use the Scrapy framework to crawl vineyard pest data; Step 2.2: Convert the vineyard pest data into a standardized vineyard pest and disease knowledge corpus using regular expressions; for semi-structured data, directly extract entities; for unstructured data, use the deep learning model Bi-LSTM-CRF for entity extraction. Step 2.3: Based on the vineyard pest and disease knowledge corpus, construct a vineyard pest and disease attribute feature knowledge graph (GPKG); the instance set is defined as a triple of <pest category, relationship, attribute feature>, and the graph database Neo4j is selected as the knowledge storage method.

4. The method for fine-grained identification of vineyard pests and diseases based on attribute feature knowledge graphs according to claim 3, characterized in that, In step 5, the training pest image features f train Or test the characteristics of pest images f test Input to the classification layer to obtain the predicted pest category; model loss Using cross-entropy loss function Sum and cosine loss function express: In the above formula, y i and These represent the true and predicted labels of the input pest image, respectively. express The predicted probability.

5. A computer storage medium, characterized in that, The computer storage medium stores a computer program; when the computer program is run on the computer, it causes the computer to execute the fine-grained identification method for vineyard pests and diseases according to any one of claims 1 to 4.

6. A computer program product, characterized in that, When the computer program product is run on a computer, the computer performs the fine-grained identification method for vineyard pests and diseases as described in any one of claims 1 to 4.

Images