Visual question answering method, system and equipment for crop diseases with external knowledge

Abstract

The present invention belongs to the technical field of natural language processing, and discloses a visual question-answering method, system and device for crop diseases that introduce external knowledge. The present invention retrieves explicit knowledge text in a crop disease knowledge base, retrieves implicit knowledge text in a pre-trained large language model, and deeply involves image information in the retrieval process of these two external knowledge texts, thereby reducing text deviation; uses a target detection model and an image subtitle generation model to process images, then uses an image feature extractor to extract image features, and uses a large language encoder model to extract text features of question text, explicit knowledge and implicit knowledge text; finally, the features of these two different modalities are input into the encoder-decoder network, and finally an answer is generated. The present invention can effectively answer questions about the visual features of crop disease images and questions involving external knowledge related to disease images.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and natural language processing, and specifically relates to a visual question-answering method, system and device for crop diseases that introduce external knowledge, which is particularly suitable for smart agriculture application scenarios. Background Art

[0002] Crop diseases are one of the main agricultural disasters, and we need to strengthen monitoring, take effective measures to prevent and control them in a timely manner, and strive to reduce yield losses. However, existing crop disease detection models, such as crop disease target detection models and crop disease classification models, can only give labels for crop diseases, but cannot provide knowledge other than images and specific prevention and control measures for different diseases. For example, for apple anthracnose at different stages, the prevention and control methods for early and late apple anthracnose are 50% thiophanate-methyl wettable powder 600-800 times diluted and 40% carbendazim glue 800 times + 40% chlorpyrifos 1500 times diluted, respectively. This task cannot be done by target detection and classification models. Traditional crop disease visual question-answering models can only reflect the visual information of crop disease images. Therefore, it is of great significance to introduce knowledge such as pathogens, chemical control agents, and crop planting methods into the visual question answering model so that the model can reflect these knowledge texts to the question askers, such as agricultural researchers, farmers and other agricultural practitioners. However, the traditional visual question answering model cannot solve the above problem. Summary of the invention

[0003] The purpose of the present invention is to propose a crop disease visual question answering method that introduces external knowledge, which can effectively answer questions about the visual features of crop disease images and external knowledge related to the disease images.

[0004] In order to achieve the above object, the present invention adopts the following technical scheme:

[0005] A crop disease visual question answering method that introduces external knowledge includes the following steps:

[0006] Step 1. Data collection, preprocessing and annotation of crop disease visual question answering based on external knowledge. Aiming at the problem that crop disease image annotation involves external knowledge, a crop disease visual question answering dataset based on external knowledge is obtained.

[0007] Step 2. Build a crop disease visual question answering model based on external knowledge, which includes an image feature extractor, an object detection model, an image caption generation model, a text feature extractor, an attention mechanism, an encoder, and a decoder;

[0008] Firstly, the features of crop disease images are extracted using a pre-trained image feature extractor, and the crop disease images are processed using a target detection model and an image caption generation model to obtain image caption information.

[0009] The image caption information contains the target box, target label and text description of the image;

[0010] Step 3. Use the crop disease image features in step 2 to retrieve explicit knowledge text of crop diseases in the crop disease knowledge base, and use the question text, image caption text and target label to retrieve implicit knowledge text of crop diseases;

[0011] Step 4. For the crop disease image features extracted in step 2, use the attention mechanism to model the dense interactions within the image modality to obtain image features with attention focus;

[0012] Step 5. Use the pre-trained text feature extractor to extract question features, image caption text features in step 2, and explicit knowledge and implicit knowledge text features of crop diseases in step 3, and encode them with the image features processed by the attention mechanism in step 4 to obtain the encoded fusion features;

[0013] Step 6. Use the fused features obtained in step 5 and input them into the decoder to output the answer.

[0014] In addition, based on the crop disease visual question answering method that introduces external knowledge, the present invention also proposes a corresponding crop disease visual question answering system that introduces external knowledge, which adopts the following scheme:

[0015] A crop disease visual question-answering system that introduces external knowledge includes the following modules:

[0016] The preprocessing module is used for data collection, preprocessing and labeling of crop disease visual question answering based on external knowledge. In view of the problem that crop disease image labeling involves external knowledge, a crop disease visual question answering dataset based on external knowledge is obtained.

[0017] A feature extraction module is used to first extract features of crop disease images using a pre-trained image feature extractor, and process the crop disease images using a target detection model and an image caption generation model to obtain image caption information;

[0018] The image caption information contains the target box, target label and text description of the image;

[0019] The external knowledge processing module is used to divide the external knowledge into explicit knowledge and implicit knowledge, and perform knowledge retrieval in the crop disease knowledge base and the pre-trained large language model to obtain the explicit knowledge and implicit knowledge of crop diseases;

[0020] The attention mechanism module is used to extract crop disease image features and use the attention mechanism to model the dense interactions within the image modality to obtain image features with attention focus;

[0021] The encoding module is used to extract question features, image caption text features, and explicit and implicit knowledge text features of crop diseases using a pre-trained text feature extractor, and encode them with the image features processed by the attention mechanism to obtain encoded fusion features;

[0022] and a prediction module, which is used to input the obtained fusion features into the decoder to output the answer;

[0023] Among them, the crop disease visual question-answering model built includes an image feature extractor, a target detection model, an image subtitle generation model, a text feature extractor, an attention mechanism, an encoder and a decoder.

[0024] In addition, based on the crop disease visual question answering method that introduces external knowledge, the present invention also proposes a computer device, which includes a memory and one or more processors. An executable code is stored in the memory. When the processor executes the executable code, it is used to implement the crop disease visual question answering method that introduces external knowledge.

[0025] The present invention has the following advantages:

[0026] As described above, the present invention relates to a visual question-answering method for crop diseases that introduces external knowledge. The method builds a visual question-answering model for crop diseases that introduces external knowledge. This model introduces explicit knowledge into a professional crop disease knowledge base. Thanks to a large amount of professional knowledge in various fields trained in the large language model GPT-3, many existing visual question-answering models introduce professional knowledge in the pre-trained GPT-3. The present invention introduces implicit knowledge of crop diseases into the pre-trained large language model GPT-3, and deeply involves image information in the retrieval process of these two external knowledge, thereby reducing text deviation. After the crop diseases are processed using the target detection model and the image caption generation model, The feature extractor is used to extract the features of the disease image, and then the image features are used to detect the explicit knowledge text. The image caption text and the target label text are used to synthesize the text prompt to retrieve the implicit knowledge text in the pre-trained large language model GPT-3. Then the large language encoder T5 model is used to extract the text features of the question text, explicit knowledge and implicit knowledge text. Compared with LSTM and small-scale Transformer models, the pre-trained large language encoder T5 model can extract text features more completely. Finally, the features of these two different modalities are input into the encoder-decoder network. This structure has been proven to be able to effectively process multi-model feature information and has been transformed into a variety of other models to generate answers. The crop disease visual question-answering model proposed in the present invention can effectively answer questions about the visual features of crop disease images and external knowledge related to disease images, and provide a reference for the acquisition of specific information on crop diseases and the formulation of prevention and control strategies in the development of smart agriculture. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A flowchart of a crop disease visual question answering method that introduces external knowledge in an embodiment of the present invention;

[0028] Figure 2 A network structure diagram of a crop disease visual question-answering model constructed in an embodiment of the present invention;

[0029] Figure 3 A diagram showing the proportion of different external knowledge types in the data set constructed in the embodiment of the present invention;

[0030] Figure 4 is the self-attention unit of the attention mechanism in the embodiment of the present invention Model diagram;

[0031] Figure 5 2 is a diagram of a stacked attention model of the attention mechanism in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0033] Example 1

[0034] like Figure 1 As shown, the crop disease visual question answering method that introduces external knowledge in this embodiment includes the following steps:

[0035] Step 1. Data collection, preprocessing and labeling of crop disease visual question answering based on external knowledge. Aiming at the problem that crop disease image labeling involves external knowledge, a crop disease visual question answering dataset based on external knowledge is obtained.

[0036] Step 1.1. Collect crop disease image data from the Internet, and perform data enhancement through various data enhancement methods such as Gaussian noise, random rotation, MSRCP and MSRCR algorithms to obtain multiple disease images.

[0037] The collected crop disease image data, for example from commonly used search sites such as Google and Baidu, covers 18 different types of crop diseases, including strawberry anthracnose, papaya anthracnose, tomato fruit cracking, tomato cottony blight, tomato anthracnose, citrus bark cracking, citrus penicillium, olive anthracnose, peanut leaf spot, pear brown rot, pear anthracnose, pear rust, mango anthracnose, apple fruit cracking, apple rot, apple water core, apple anthracnose and jujube anthracnose.

[0038] After collecting crop disease image data, we select high-quality images with complete overall crops and clear features of crops and lesion edge features, and perform data enhancement, ultimately obtaining thousands of disease images.

[0039] Step 1.2. Design question texts based on the features of the crop disease image itself and the knowledge outside the image.

[0040] The feature types of crop disease images themselves include: disease type, crop type, shape, color, state, texture, quantity, location, and size of lesions and fruits.

[0041] There are 13 types of knowledge other than crop disease images, including: pathogens, prevention and control measures, disease environment, sowing period, disease-resistant varieties, chemical control agents, chemical prevention, nutrient requirements, recommended planting environment, disease stage, recommended planting method, disease transmission mode and prevention and control measures at different periods.

[0042] The proportion of different types of external knowledge, such as Figure 3 shown.

[0043] Depend on Figure 3It is not difficult to see that problems related to the disease environment, chemical control agents, disease transmission methods and prevention and control measures at different periods account for a relatively large proportion, while problems related to the recommended planting environment and nutrient requirements are relatively few.

[0044] A series of question texts are annotated for different areas of each crop disease image; then the question texts are augmented by random punctuation replacement, keyword replacement, and changing the grammar of the question text.

[0045] Finally, on average, each image is annotated with multiple (for example, 8 in this embodiment) question texts focusing on the image content and multiple (for example, 8 in this embodiment) question texts focusing on external knowledge, thereby constructing tens of thousands of image-question text pairs, and the data set is divided into training set, validation set, and test set in a ratio of 8:1:1.

[0046] Step 1.3. Annotate answers for each question text; each question text is annotated with multiple (for example, 10 in this embodiment) answers with different confidence levels. Among these answers, items with high confidence levels and large numbers are selected as correct answers and provided to the external knowledge-based crop disease visual question-answering model built in the following step 2 for learning.

[0047] Step 2. Build a crop disease visual question answering model based on external knowledge, such as Figure 2 shown.

[0048] The crop disease visual question-answering model built includes an image feature extractor, an object detection model, an image caption generation model, a text feature extractor, an attention mechanism, an encoder, and a decoder.

[0049] like Figure 2 As shown in the figure, the processing flow of the crop disease visual question answering model is as follows:

[0050] For the input crop disease image, the pre-trained image caption generation model OPD-Transformer is first used to generate image caption text; then the target detection model Faster-RCNN is used to detect the disease target in the image, and the disease target location information and disease label text are obtained; then the image feature extractor Inception-v4 model is used to extract the image features of the disease target location, and then the partial image features of the disease target are used to retrieve the disease knowledge text in the crop disease knowledge base "Chinese Crop Diseases Third Edition" as explicit knowledge, and the disease label text and disease image caption text are combined and sent to the pre-trained large language model GPT-3 model to retrieve implicit knowledge; then the pre-trained large language encoder T5 model is used to extract the question text, crop disease explicit knowledge and implicit knowledge text features respectively, and these three text features and the image features of the disease target location are sent to the encoder-decoder to finally generate the answer text.

[0051] The specific execution process of each module of the model is described in detail below.

[0052] Firstly, the features of crop disease images are extracted using a pre-trained image feature extractor, and the crop disease images are processed using a target detection model and an image caption generation model to obtain image caption information.

[0053] The image caption information contains the target box, target label and text description of the image.

[0054] Step 2.1. The input of the visual question answering model is divided into two modal data: crop disease images and disease-related question texts; the image modal data is represented as , the text modality data of the question is represented as , and express the answer as .

[0055] Among them The diseased image is represented as , No. The question text and its answer are represented as and , the overall crop disease dataset is represented as , is the number of image-question-answer triplets.

[0056] Step 2.2. The image feature extractor uses the pre-trained Inception-v4 model and extracts the image features within the target detection box. The Inception-v4 model is represented as , and its encoded features are expressed as .

[0057] Step 2.3. Represent the pre-trained object detection model Faster-RCNN as , the crop disease images enter In, get There are two output results:

[0058] Coordinate information of the target detection box And the category label of the crop or disease in the detection frame .

[0059] in is the number of target detection boxes, Represents the coordinate information of the jth target detection box, Represents the crop species label or crop disease category label of the j-th target detection box.

[0060] To supplement the category labels of crops or diseases , the pre-trained image caption generation model ODP-Transformer is used to process the diseased image, and the pre-trained image caption generation model ODP-Transformer is expressed as .

[0061] The generated caption text of crop disease images is represented as .

[0062] Step 3. Divide the external knowledge into explicit knowledge and implicit knowledge, and then use the different information obtained in step 2 to perform knowledge retrieval according to different types of knowledge to obtain explicit knowledge and implicit knowledge of crop diseases.

[0063] The explicit knowledge and implicit knowledge texts come from "China's Crop Diseases, Third Edition" and the pre-trained large language model GPT-3 model, respectively. Among them, "China's Crop Diseases, Third Edition" is used as the explicit knowledge base of crop diseases.

[0064] Step 3.1. Divide external knowledge into explicit knowledge and implicit knowledge, and use different methods to retrieve the two types of knowledge. The explicit knowledge of crop diseases uses object-centric image features. The implicit knowledge of crop diseases is obtained using the pre-trained large language model GPT-3.

[0065] Step 3.2. Use the "China Crop Diseases 3rd Edition" as the explicit knowledge base of crop diseases and extract a set of candidate knowledge for each crop disease , Each entry in consists of an entity and a description.

[0066] For example, "Olive Anthracnose": "The pathogen overwinters in the form of mycelium on infected branches, or in the form of mycelium and conidia on infected dead branches, fallen leaves, rotten fruits and other diseased and residual tissues. In the rainy season from March to April in the spring of the following year, the overwintering pathogen produces conidia, which are spread by wind, rain or insects to cause initial infection."

[0067] In order to Select the item that is most relevant to the image information. The entries in are all formatted as "'entity' is 'description'", represented as , will have a high correlation Items as diseased images Explicit knowledge , using FAISS to accelerate the retrieval speed of explicit knowledge in the formula.

[0068] Step 3.3. Use the pre-trained large language model GPT-3 as the implicit knowledge base.

[0069] The text prompts input into GPT-3 are standardized as follows: : "Text: {image caption description text} + {class label of target detection} + question: {question text}", the resulting set of candidate outputs is represented as .

[0070] Using candidate output Redesign the text prompt to "{question}+{ }, this is because: ", this text prompt is fed back into the GPT-3 model to obtain a complete description of the crop disease , implicit knowledge It is expressed as:

[0071] ;in, represents the number of candidate knowledge, u ∈ [ 1 , U ] .

[0072] Step 4. For the crop disease image features extracted in step 2, use the attention mechanism to model the dense interactions within the image modality to obtain image features with attention focus.

[0073] Step 4.1. Feature the crop disease image Through three different matrix projections, the transformed feature matrix is ​​obtained , and ,in , , for a given and Yes, compute the scaled dot product:

[0074] ;

[0075] in, represents the number of image features input to the scaled click model, express , Dimension.

[0076] Depend on The scaled dot products are combined in parallel to form a self-attention unit ,like Figure 4 shown.

[0077] p = H A ( Q U , K E , V A ) = [ s c l t 1 , s c l t 2 , . . . , s c l t i , . . . , s c l t h ] W o ;

[0078] .

[0079] in, represents a self-attention unit, i ∈ [ 1 , h ] , , , Represent three different mapping matrices multiplied by the input image features, , Indicates the number of parallel scaled dot product models.

[0080] Step 4.2. The stacking of layer self-attention units is called Stacking Self-Attention Model, );Will Each layer is represented as ,like Figure 5 shown.

[0081] in, is the first layer attention unit of the self-attention mechanism, and the second layer self-attention unit is expressed as The input is , represents the input image, Represents the encoded image features.

[0082] Define each layer The output is expressed as , No. layer The input and output are expressed as: .

[0083] Step 5. Use the pre-trained text feature extractor to extract question features, the image caption text features in step 2, and the explicit and implicit knowledge text features of crop diseases in step 3, and encode them with the image features processed by the attention mechanism in step 4 to obtain the encoded fusion features.

[0084] Step 5.1. Encode and decode the input multimodal data using the encoder and decoder architecture. The input multimodal data are explicit knowledge text, implicit knowledge text, image features within the image object detection box, and question text.

[0085] Step 5.2. The coordinate information of the target detection box As position information and image region features after attention mechanism , respectively input into two different fully connected layer networks and In , all regional features of an image are spliced.

[0086] Finally, the concatenated feature matrix is ​​input into the decoder In the above example, we obtain the object-centric image feature encoding ;

[0087] .

[0088] in, , Indicates The image features of the object detection box after the attention mechanism, is the coordinate information of the input target detection box, It is a matrix concatenation operation along one dimension.

[0089] Step 5.3. External knowledge is divided into explicit knowledge and implicit knowledge. Use explicit knowledge The entity and description of are reset to "entity: {entity} + description: {description}", and the reset explicit knowledge is represented as .

[0090] in k ∈ [ 1 , K ] , represents the number of explicit knowledge;

[0091] For implicit knowledge, the candidate answers and a complete description of external knowledge of crop diseases acquired through GPT-3 Combining tacit knowledge into reset , represented as: “Candidate answer: {answer} + evidence: {complete description}”.

[0092] For question text, convert it into a text prompt , as the problem input to the model.

[0093] Step 5.4. Use the pre-trained large language encoder T5 model to train explicit knowledge , tacit knowledge and text prompts Encoding, the large language encoder T5 model is represented as , the text embedding dimension is .

[0094] The encoding of the two kinds of knowledge and textual prompts is represented as:

[0095] , , ;

[0096] in, Represents explicit knowledge encoding, Represents implicit knowledge encoding, Indicates the text prompt encoding.

[0097] Step 6. Concatenate the encoded image and text features, and then use the decoder to output the answer.

[0098] Encoding explicit knowledge , implicit knowledge encoding , object-centric image feature encoding and text prompt encoding Concatenate along the first dimension and then input into the decoder The output answer is :

[0099] ;

[0100] in, express explicit knowledge encoding, express The implicit knowledge encoding Represents the operation of concatenating features. The answer to the model output Calculating cross entropy loss :

[0101] ;

[0102] in, is the length of the answer text, Indicates the current answer text length, To mark the answer, To predict the answer, To cause losses The model parameters.

[0103] Then Backpropagation to update model parameters of the crop disease visual question answering model .

[0104] After the crop disease visual question-answering model is trained, the model gives corresponding question-answering results based on the input questions.

[0105] For example, for the input apple anthracnose image and questions raised by agricultural practitioners, such as "What is the pathogen that infects the crops in the image?", "What is the chemical agent for preventing and treating the current disease of the crops in the image?" and "What is the most effective prevention and control measure for the disease in the image?", the crop disease visual question answering model gives the question and answer results, which are "Colletocystis gloeosporioides", "40% carbendazim glue 800 times + 40% chlorpyrifos 1500 times solution" and "fruit bagging".

[0106] In addition, in order to prove the feasibility and superiority of the method of the present invention, a comparative experiment was conducted with existing methods (12 models including the model proposed in the present invention were compared on the public dataset OK-VQA). The descriptions of each model are as follows:

[0107] Q only: a simplified or specific query model used in information retrieval or question-answering systems that only processes query input; MLP (Multi-layer Perceptron): a basic neural network that contains multiple levels of linear transformations and nonlinear activation functions; BAN+AN (Bidirectional Attention Network + Adaptive Network): a model used for natural language processing tasks that combines bidirectional attention and adaptive mechanisms; MUTAN (Multimodal Bilinear Attention Network): a model used to process multimodal data that fuses information from different modalities through a bilinear attention mechanism. ConceptBERT: an extended model based on BERT that incorporates additional conceptual knowledge to enhance language understanding; KRISP (Knowledge Enhanced Image Semantic Parsing): a model for image semantic parsing tasks that enhances parsing capabilities by integrating external knowledge bases; VRR (Visual Relational Reasoning): a model that focuses on visual relationship reasoning in images, capable of identifying and analyzing entities and their relationships in images. PICa-Base / PICa-Full (Base / Full Model for Partial Image-to-Text Matching): Models for image-to-text matching tasks, Base is the basic version, Full is the full version, with stronger matching capabilities; KAT (Knowledge-Augmented Transformer): A model that combines an external knowledge base and a transformer architecture to enhance the effects of natural language processing tasks; REVIVE (Retrieval-based Visual-Language Model): A visual-language model that fuses visual and language information through a retrieval mechanism for a variety of cross-modal tasks.

[0108] The experimental results are shown in Table 1, where the accuracy evaluation index is the ratio of all positive samples to all samples. The larger the value of the accuracy evaluation index, the stronger the performance of the model in the visual question answering task based on external knowledge.

[0109] Table 1 Performance comparison of various models

[0110]

[0111] By comparison, it is not difficult to find that models that lack external knowledge sources, such as Q only, MLP and MUTAN, have low accuracy. BAN+AN and ConceptBERT, which introduce Wikipedia and ConceptNet knowledge bases, have improved their accuracy by 4.9% and 7.3% respectively. KRISP improves its accuracy by 4.7% by summarizing the knowledge of Wikipedia and ConceptNet. VRR, PICa-Base and PICa-Full have similar accuracy rates, all of which use large language models for knowledge retrieval. KAT and REVIVE combine large language models with fixed knowledge bases, with the highest accuracy rate reaching 56.6%. The model proposed in the present invention has the highest accuracy, which is 3.2% higher than the previous best model, and the number of model parameters is effectively controlled.

[0112] The comparison results show that the present invention has achieved the best prediction results, and its accuracy is the best compared with other models, and its model size is maintained within a controllable range. The model of the present invention has achieved the best balance between accuracy and model parameter quantity.

[0113] Example 2

[0114] This embodiment 2 describes a crop disease visual question answering system that introduces external knowledge. This system is based on the same inventive concept as the crop disease visual question answering method that introduces external knowledge described in the above embodiment 1.

[0115] A crop disease visual question-answering system that introduces external knowledge includes the following modules:

[0116] The preprocessing module is used for data collection, preprocessing and labeling of crop disease visual question answering based on external knowledge. In view of the problem that crop disease image labeling involves external knowledge, a crop disease visual question answering dataset based on external knowledge is obtained.

[0117] A feature extraction module is used to first extract features of crop disease images using a pre-trained image feature extractor, and process the crop disease images using a target detection model and an image caption generation model to obtain image caption information;

[0118] The image caption information contains the target box, target label and text description of the image;

[0119] The external knowledge processing module is used to divide the external knowledge into explicit knowledge and implicit knowledge, and perform knowledge retrieval in the crop disease knowledge base and the pre-trained large language model to obtain the explicit knowledge and implicit knowledge of crop diseases;

[0120] The attention mechanism module is used to extract crop disease image features and use the attention mechanism to model the dense interactions within the image modality to obtain image features with attention focus;

[0121] The encoding module is used to extract question features, image caption text features, and explicit and implicit knowledge text features of crop diseases using a pre-trained text feature extractor, and encode them with the image features processed by the attention mechanism to obtain encoded fusion features;

[0122] and a prediction module, which is used to input the obtained fusion features into the decoder to output the answer;

[0123] Among them, the crop disease visual question-answering model built includes an image feature extractor, a target detection model, an image subtitle generation model, a text feature extractor, an attention mechanism, an encoder and a decoder.

[0124] It should be noted that, in the crop disease visual question-answering system in this embodiment 2, the implementation process of the functions and effects of each functional module is specifically described in the implementation process of the corresponding steps in the method in the above embodiment 1, and will not be repeated here.

[0125] Example 3

[0126] This embodiment 3 describes a computer device, which includes a memory and one or more processors. The memory stores executable code. When the processor executes the executable code, it is used to implement the steps of the crop disease visual question answering method that introduces external knowledge in the above embodiment 1.

[0127] In this embodiment, the computer device is any device or apparatus with data processing capability, which will not be described in detail here.

[0128] Example 4

[0129] This embodiment 4 describes a computer-readable storage medium on which a program is stored. When the program is executed by a processor, it is used to implement the steps of the crop disease visual question-answering method that introduces external knowledge in the above embodiment 1.

[0130] The computer-readable storage medium may be an internal storage unit of any device or apparatus with data processing capabilities, such as a hard disk or memory, or an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc., equipped on the device.

[0131] Of course, the above description is only a preferred embodiment of the present invention, and the present invention is not limited to the above embodiments. It should be noted that all equivalent substitutions and obvious deformation forms made by any technician familiar with the field under the guidance of this specification fall within the essential scope of this specification and should be protected by the present invention.

Claims

1. A visual question answering method for crop diseases that introduces external knowledge, characterized in that: The steps include: Step 1. Data collection, preprocessing and annotation of crop disease visual question answering based on external knowledge. Aiming at the problem that crop disease image annotation involves external knowledge, a crop disease visual question answering dataset based on external knowledge is obtained. Step 2. Build a crop disease visual question answering model based on external knowledge, which includes an image feature extractor, an object detection model, an image caption generation model, a text feature extractor, an attention mechanism, an encoder, and a decoder; Firstly, the features of crop disease images are extracted using a pre-trained image feature extractor, and the crop disease images are processed using a target detection model and an image caption generation model to obtain image caption information. The image caption information contains the target box, target label and text description of the image; Step 3. Use the crop disease image features in step 2 to retrieve explicit knowledge text of crop diseases in the crop disease knowledge base, and use the question text, image caption text and target label to retrieve implicit knowledge text of crop diseases; Step 4. For the crop disease image features extracted in step 2, use the attention mechanism to model the dense interactions within the image modality to obtain image features with attention focus; Step 5. Use the pre-trained text feature extractor to extract question features, image caption text features in step 2, and explicit knowledge and implicit knowledge text features of crop diseases in step 3, and encode them with the image features processed by the attention mechanism in step 4 to obtain the encoded fusion features; Step 6. Use the fused features obtained in step 5 and input them into the decoder to output the answer.

2. The crop disease visual question answering method with external knowledge introduction according to claim 1 is characterized in that: The step 1 is specifically as follows: Step 1.

1. Collect crop disease image data from the Internet, select high-quality images with complete crop structure and clear crop and lesion edge features, perform data enhancement, and obtain multiple disease images; Step 1.

2. Design question texts based on the features of the crop disease image itself and the knowledge outside the image; Label different regions of each crop disease image with multiple question texts; Then, data augmentation is performed on the question text by using random punctuation replacement, keyword replacement, and changing the grammar of the question text; Finally, we annotate each image with multiple question texts focusing on the image content and multiple question texts focusing on external knowledge, thereby constructing image-question text pair data and dividing it into training set, validation set, and test set. Step 1.

3. Label the answer for each question text; each question text is labeled with multiple answers with different confidence levels. Among these answers, the items with high confidence and large number are selected as the correct answers and provided to the model for learning.

3. The crop disease visual question answering method with external knowledge introduction according to claim 1 is characterized in that: The step 2 is specifically as follows: Step 2.

1. The input of the visual question answering model is divided into two modal data: crop disease images and disease-related question texts; the image modal data is represented as , the text modality data of the question is represented as , and express the answer as ; No. The diseased image is represented as , No. The question text and its answer are represented as and , the overall crop disease dataset is represented as ,in is the number of image-question-answer triplets; Step 2.

2. The image feature extractor uses the pre-trained Inception-v4 model and extracts the image features within the target detection frame. The pre-trained Inception-v4 model is represented as , and its encoded features are expressed as ; Step 2.

3. Represent the pre-trained object detection model Faster-RCNN as , the crop disease images enter In, get There are two output results: Coordinate information of the target detection box And the category label of the crop or disease in the detection frame ; in is the number of target detection boxes, Represents the coordinate information of the jth target detection box, Represents the crop species label or crop disease category label of the jth target detection frame; The pre-trained image caption generation model ODP-Transformer is used to process the diseased image, and ODP-Transformer is represented as , the generated caption text of crop disease image is represented as .

4. The crop disease visual question answering method with external knowledge introduction according to claim 1, characterized in that: The step 3 is specifically as follows: Step 3.

1. Divide external knowledge into explicit knowledge and implicit knowledge, and use different methods to retrieve the two types of knowledge; the explicit knowledge of crop diseases uses object-centric image features Obtain,the hidden knowledge of crop diseases using the pre-trained large language model GPT-3; Step 3.

2. Using the crop disease explicit knowledge base, extract a set of candidate knowledge for each crop disease , Each entry in consists of an entity and a description; Will All entries in the format "'entity' is 'description'", represented as , will have a high correlation Items as diseased images Explicit knowledge , using FAISS to accelerate the retrieval speed of explicit knowledge in formulas; Step 3.

3. Use the pre-trained large language model GPT-3 as the implicit knowledge base; The text prompts input into GPT-3 are standardized as follows: : "text: {image caption description text} + {class label for target detection} + question: {question text}", the resulting set of candidate outputs is represented as ; Using candidate output Redesign the text prompt to "{question} + { }, this is because: "The text prompt is input into the GPT-3 model again to obtain a complete description of crop diseases. , implicit knowledge It is expressed as: ;in, represents the number of candidate knowledge, .

5. The crop disease visual question answering method with external knowledge introduction according to claim 1 is characterized in that: The step 4 is specifically as follows: Step 4.

1. Feature the crop disease image Through three different matrix projections, the transformed feature matrix is ​​obtained , and ,in , , for a given and Yes, compute the scaled dot product: ； in, represents the number of image features input to the scaled click model, express , Dimensions; Depend on The scaled dot products are combined in parallel to form a self-attention unit : ； ； in, represents a self-attention unit, , , , Represent three different mapping matrices multiplied by the input image features, , Indicates the number of parallel scaled dot product models; Step 4.

2. Layer self-attention units are stacked as stacked attention models ; Will Each layer is represented as ; The input is ; in, Represents the encoded image features; Define each layer The output is expressed as , No. layer The input and output are expressed as: .

6. The crop disease visual question answering method with external knowledge as claimed in claim 1, characterized in that: The step 5 is specifically as follows: Step 5.

1. Encode and decode the input multimodal data using the encoder and decoder architecture, where the input multimodal data are crop disease explicit knowledge text, crop disease implicit knowledge text, object-centric image features, and question text; Step 5.

2. The coordinate information of the target detection box As position information and image region features after attention mechanism , respectively input into two different fully connected layer networks and In the above example, all regional features of an image are concatenated to obtain the object-centered image feature encoding. ; in, is the number of target detection boxes; Represents the coordinate information of the jth target detection box, Indicates Image features of the object detection box after the attention mechanism; Step 5.

3. Use explicit knowledge The entity and description of are reset to "entity: {entity} + description: {description}", and the reset explicit knowledge is represented as ,in , represents the number of explicit knowledge; For implicit knowledge, the candidate answers and a complete description of external knowledge of crop diseases acquired through GPT-3 Combining tacit knowledge into reset , represented as: "Candidate answer: {answer} + evidence: {complete description}"; For question text, convert it into a text prompt , as the problem input to the model; Step 5.

4. Use the pre-trained large language encoder T5 model to train explicit knowledge , tacit knowledge and text prompts Encoding, the large language encoder T5 model is represented as , the text embedding dimension is ; The encoding of the two kinds of knowledge and textual prompts is represented as: ， ， ； in, Represents explicit knowledge encoding, Represents implicit knowledge encoding, Indicates the text prompt encoding.

7. The crop disease visual question answering method with external knowledge as claimed in claim 6, characterized in that: The step 6 is specifically as follows: Encoding explicit knowledge , implicit knowledge encoding , object-centric image feature encoding and text prompt encoding Concatenate along the first dimension and then input into the decoder The output answer is : ； in, express explicit knowledge encoding, express The implicit knowledge encoding Represents an operation of concatenating features.

8. The crop disease visual question answering method with external knowledge as claimed in claim 7, characterized in that: In step 6, the answer to the model output Calculating cross entropy loss : ； in, is the longest answer text length, Indicates the current answer text length, To mark the answer, To predict the answer, To cause losses Model parameters of Then Backpropagation to update model parameters of the crop disease visual question answering model .

9. A visual question-answering system for crop diseases that introduces external knowledge, characterized in that: Includes the following modules: The preprocessing module is used for data collection, preprocessing and labeling of crop disease visual question answering based on external knowledge. In view of the problem that crop disease image labeling involves external knowledge, a crop disease visual question answering dataset based on external knowledge is obtained. A feature extraction module is used to first extract features of crop disease images using a pre-trained image feature extractor, and process the crop disease images using a target detection model and an image caption generation model to obtain image caption information; The image caption information contains the target box, target label and text description of the image; The external knowledge processing module is used to divide the external knowledge into explicit knowledge and implicit knowledge, and perform knowledge retrieval in the crop disease knowledge base and the pre-trained large language model to obtain the explicit knowledge and implicit knowledge of crop diseases; The attention mechanism module is used to extract crop disease image features and use the attention mechanism to model the dense interactions within the image modality to obtain image features with attention focus; The encoding module is used to extract question features, image caption text features, and explicit and implicit knowledge text features of crop diseases using a pre-trained text feature extractor, and encode them with the image features processed by the attention mechanism to obtain encoded fusion features; and a prediction module, which is used to input the obtained fusion features into the decoder to output the answer; Among them, the crop disease visual question-answering model built includes an image feature extractor, a target detection model, an image subtitle generation model, a text feature extractor, an attention mechanism, an encoder and a decoder.

10. A computer device comprising a memory and one or more processors; an executable code is stored in the memory; characterized in that: When the processor executes the executable code, it is used to implement the crop disease visual question answering method introducing external knowledge as described in any one of claims 1 to 8.

Images