A crop disease visual question answering method, system, computer device and medium

Abstract

The present invention belongs to the technical field of computer vision and natural language processing, and discloses a visual question-answering method, system, computer device and medium for crop diseases. The present invention first uses a feature extractor to extract feature vectors of images and questions, then deeply fuses the information of these two modes, and then uses a deviation balance strategy to process the fused feature vectors, and finally uses a classifier to process the generated feature vectors to obtain the answer to the question. The present invention can solve the problem that the existing visual question-answering model cannot obtain multi-attribute visual information of crop disease images and judge the stage of disease occurrence according to this information. The crop disease visual question-answering model based on prior knowledge proposed in the present invention, by separating the visual question-answering task and the deviation balance task, is conducive to solving the problem that the single-peak deviation misleads the model training and causes the model to be unable to produce a true description according to the crop disease image, thereby improving the model's prediction accuracy for disease images in different conditions.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and natural language processing, and in particular relates to a crop disease visual question-answering method, system, computer equipment and medium. Background Art

[0002] Crop diseases can cause huge yield losses. Relying on the various visual attribute information of crop diseases can determine the specific stage of the disease. Formulating treatment methods according to the different stages of crop diseases is crucial to ensure agricultural productivity. However, existing deep learning models cannot obtain the corresponding multiple visual attribute information based on crop disease images.

[0003] To solve this problem, a visual question answering model is used to obtain text descriptions of multiple visual attribute information in disease images. That is, a visual question answering model is trained using a crop disease dataset, image and question features are processed, and then an answer generator is trained using the fused features to complete the visual question answering task. However, the visual question answering model has an inherent problem, that is, the model will learn the statistical laws of the answers in the dataset during the training process, resulting in the visual question model answering questions out of touch with the real information in the image, reducing the performance of the model in analyzing new scene images. This problem is called the unimodal deviation problem. When it is manifested in the crop disease visual question answering task, the visual features of individual disease images are different from the visual features of most of the same diseases. For example, the color of the center of most apple anthracnose lesions is brown. When a red lesion image appears, the traditional visual question answering model will still answer brown according to the statistical law, rather than answering correctly according to the visual features of the image, resulting in the model being unable to generate a true description of the crop disease image, thereby reducing the prediction accuracy of the model. Summary of the invention

[0004] The purpose of the present invention is to propose a crop disease visual question answering method to solve the problem that the existing visual question answering model cannot obtain multi-attribute visual information of crop disease images and judge the stage of disease occurrence according to this information. The method effectively solves the problem that the unimodal deviation misleads model training and causes the model to be unable to produce a true description according to the crop disease images, which is widely present in visual question answering tasks. It can improve the model's prediction accuracy for disease images in different conditions.

[0005] In order to achieve the above object, the present invention adopts the following technical scheme:

[0006] A crop disease visual question answering method, characterized by comprising the following steps:

[0007] Step 1. Collect, preprocess and label crop disease visual question answering data;

[0008] The crop disease prior knowledge VQA data is mixed into the conventional crop disease VQA data set to obtain a crop disease visual question answering data set containing prior knowledge. The method of training the model using this data set is called informed learning.

[0009] Step 2. Build a crop disease visual question answering model based on prior knowledge, which includes image visual feature extractor, text feature extractor, attention mechanism, multimodal fusion model, bias balance strategy and classifier;

[0010] Firstly, the image visual feature extractor and question text feature extractor are used to extract the visual features and question text features of each image in the crop disease visual question answering dataset containing prior knowledge.

[0011] Step 3. For the obtained image visual features and question text features, use the attention mechanism to process the dense interactions within the image and between the image and the question text, and obtain semantically aligned image visual features and question text features;

[0012] Step 4. Use the multimodal fusion model to deeply fuse the semantically aligned image visual features and question text features obtained in step 3 to obtain a fused feature vector.

[0013] Step 5. Use the bias balancing strategy to process the fused feature vector to reduce the unimodal bias learned by the crop disease visual question answering model and guide the model to answer correctly according to the visual features of the disease image;

[0014] Step 6. Train a classifier to classify the low unimodal deviation features obtained after the deviation balance strategy in step 5, and finally obtain the visual question answering results of crop diseases.

[0015] In addition, based on the crop disease visual question answering method, the present invention also proposes a corresponding crop disease visual question answering system, which adopts the following scheme:

[0016] A crop disease visual question-answering system, comprising the following modules:

[0017] The preprocessing module collects, preprocesses and labels the crop disease visual question-answering data;

[0018] The crop disease prior knowledge VQA data is mixed into the conventional crop disease VQA data set to obtain a crop disease visual question answering data set containing prior knowledge. The method of training the model using this data set is called informed learning.

[0019] The feature extraction module uses an image visual feature extractor and a text feature extractor to respectively extract the visual features of each image and the question text features in the crop disease visual question answering dataset containing prior knowledge;

[0020] The attention module uses the attention mechanism to process the dense interactions within the image and between the image and the question text, respectively, based on the obtained image visual features and question text features, to obtain semantically aligned image visual features and question text features;

[0021] The multimodal fusion module is used to obtain a fused feature vector by deeply fusing the semantically aligned image visual features and question text features obtained by the multimodal fusion model.

[0022] The bias balance module is used to process the fused feature vector using the bias balance strategy to reduce the single-peak bias learned by the visual question answering model and guide the model to answer correctly according to the visual features of the disease image;

[0023] And the classification module is used to train a classifier to classify the low unimodal deviation features obtained after the deviation balance strategy is processed, and finally obtain the visual question answering results of crop diseases.

[0024] In addition, based on the crop disease visual question answering method, the present invention also proposes a computer device, which includes a memory and one or more processors. An executable code is stored in the memory. When the processor executes the executable code, the steps of the crop disease visual question answering method are implemented.

[0025] In addition, based on the crop disease visual question answering method, the present invention also proposes a computer-readable storage medium on which a program is stored. When the program is executed by a processor, it is used to implement the steps of the above-mentioned crop disease visual question answering method.

[0026] The present invention has the following advantages:

[0027] As described above, the present invention relates to a crop disease visual question answering method, system, computer device and medium. The present invention first uses a feature extractor to extract feature vectors of an image and a question, then deeply fuses the information of the two modalities, then uses a bias balance strategy to process the fused feature vector, and finally uses a classifier to process the feature vector generated in the above stages to obtain the answer to the question. The present invention can solve the problem that the existing visual question-answering model cannot obtain multi-attribute visual information of crop disease images and judge the stage of disease occurrence according to this information. The crop disease visual question-answering model based on prior knowledge proposed by the present invention is conducive to solving the problem of single-peak deviation misleading model training by separating the visual question-answering task and the deviation balance task. In the crop disease visual question-answering task, the visual features of individual disease images are different from the visual features of most of the same diseases. For example, the color of the center of the lesion of most apple anthracnose is brown. When a red lesion image appears, the traditional visual question-answering model will still answer brown according to statistical laws, instead of answering correctly according to the visual features of the image. The method of the present invention can use the deviation balance strategy to obtain the correct answer, so that the visual question-answering model can generate a true description according to the crop disease image, thereby improving the model's prediction accuracy for disease images in different conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flow chart of a visual question-answering method for crop diseases in an embodiment of the present invention;

[0029] Figure 2 is an overall network structure diagram of a crop disease visual question answering model based on prior knowledge in an embodiment of the present invention;

[0030] Figure 3 It is the basic unit that constitutes the attention mechanism in the embodiment of the present invention. and Model diagram of Figure 3 (a) in Model diagram, Figure 3 (b) in Model diagram of

[0031] Figure 4 In the embodiment of the present invention, and The attention mechanism flow chart of the composition;

[0032] Figure 5 This is a flow chart of using the deviation balancing strategy to reduce the single peak deviation of the model in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0034] Example 1

[0035] like Figure 1 As shown, this embodiment describes a method for visual question answering of crop diseases based on prior knowledge, firstly, a feature extractor is used to extract image visual features and question text features; an attention mechanism is used to process the dense interaction within the image and between the image and the question text; a multimodal fusion model is used to deeply fuse the two modal features; a deviation balance strategy is used to reduce the unimodal deviation learned by the model; and a classifier is trained to classify the above obtained features. The present invention can not only solve the problem that the existing visual question answering model cannot obtain multi-attribute visual information of crop disease images and judge the stage of disease occurrence according to this information, but also effectively solves the problem that the unimodal deviation misleads model training, which is widely present in visual question answering tasks.

[0036] The crop disease visual question answering method based on prior knowledge in this embodiment includes the following steps:

[0037] Step 1. Collect, preprocess and label crop disease visual question answering data.

[0038] The crop disease prior knowledge visual question answering (VQA) data is mixed into the conventional crop disease VQA dataset to obtain a crop disease visual question answering dataset containing prior knowledge. The method of using this dataset to train a model is called informed learning.

[0039] Step 1.1. Search the original disease image data, which contains multiple categories.

[0040] In this embodiment, the original disease image data comes from, for example, Google and Baidu searches, and the original disease image data contains 18 categories, namely: strawberry anthracnose, papaya anthracnose, tomato fruit cracking disease, tomato cottony blight, tomato anthracnose, citrus bark cracking disease, citrus penicillium, olive anthracnose, peanut leaf spot, pear brown rot, pear anthracnose, pear rust, mango anthracnose, apple fruit cracking disease, apple core rot, apple water core disease, apple anthracnose and jujube anthracnose.

[0041] High-quality image data with higher pixels and clear lesion edges and internal visual features are selected from the original disease image data, and then data enhancement is performed on the selected high-quality images.

[0042] The enhancement methods include the use of Gaussian noise addition algorithm, random angle rotation, Multi-Scale Retinex with Chromaticity Preservation (MSRCP) algorithm and Multi-Scale Retinex with Color Restoration (MSRCR) algorithm.

[0043] Next, select selected images of equal proportion for each disease in the enhanced image. These selected images have clear imaging, standard fruit and lesion features, and complete targets. Further, mark each selected image with several annotation boxes to highlight important targets in the image (such as lesion parts and fruit parts).

[0044] Finally, the labeled images are mixed with conventional disease image data and added to the model training process. This process is known as informed learning. The labeled images are added as prior knowledge to the informed learning. Through this step, multiple disease images are obtained.

[0045] Step 1.2. Design a series of visual attributes of different image focus angles, including disease type, crop type, shape, color, state, texture, quantity, location, and size of lesions and fruits.

[0046] For the image data in the conventional crop disease VQA dataset, a variety of questions are raised based on the above visual attributes. These questions are evenly distributed to each crop disease image as its question text annotation data.

[0047] For images of informed learning tasks, we design question texts for each feature in the annotation box one by one, and present its visual features to the model to the greatest extent; then we perform data augmentation on the question texts;

[0048] The methods of data augmentation here include random punctuation replacement, keyword replacement, and changing the grammar of the question text.

[0049] Finally, we annotate each disease image or image annotation box with several questions focusing on different visual attributes, and construct tens of thousands of image-question text and image annotation box-question text pair data.

[0050] Step 1.3. Label the answer to each question.

[0051] Each question is marked with several (for example, 10) answers with different confidence levels. Among these answers, the ones with high confidence levels and large numbers are selected as the correct answers and provided to the model for learning.

[0052] Step 2. Build a crop disease visual question answering model based on prior knowledge, such as Figure 2 shown.

[0053] Build an Informed-Learning guided Visual Question Answering Model of Crop Disease (ILCD) based on prior knowledge, which includes image visual feature extractor, text feature extractor, attention mechanism, multimodal fusion model, bias balance strategy and classifier.

[0054] The image visual feature extractor can use the Inception-v4 model, and the text feature extractor can use the Long Short-Term Memory (LSTM) network.

[0055] The signal processing flow in the crop disease visual question answering model ILCD based on prior knowledge is as follows:

[0056] First, the pre-trained image visual feature extractor Inception-v4 model and the pre-trained text feature extractor LSTM are used to extract feature vectors of the two modalities of image and question text; then they are input into a common attention mechanism to process the dense modeling between and within the two modalities of image visual features and question text features, and then they are input into a multimodal fusion model to deeply fuse the multimodal features processed by the common attention mechanism, and then the deviation balancing strategy proposed in the present invention is used to process the unimodal deviation, that is, the question text features are used as masks to guide the visual question answering model to answer according to the real visual information of the image, and finally a classifier is used to obtain the output answer.

[0057] Specifically, the image visual feature extractor and text feature extractor are first used to extract the visual features and question text features of each image, and then they are respectively input into the crop disease visual question answering model based on prior knowledge.

[0058] Step 2.1. The input of the visual question answering model is divided into two modal data: crop disease images and disease-related question texts. The image modal data is represented as , among which The image is represented as ; Represent the question text modal data as , a disease image The question text is represented as , the annotated answer text is represented as , The corresponding answer text is expressed as ; The overall crop disease dataset is represented as , which contains multiple - - right.

[0059] Step 2.2. Normalize each image input to the model to a size of 640*640; to handle variable problem lengths , use the 0-filling method to make each question vector equal in size.

[0060] Step 2.3. Given a pre-trained image encoder and text encoder .

[0061] use and The crop disease images and question texts are encoded respectively to obtain the features of the corresponding modalities, where the visual features of the image are represented as , the question text feature is expressed as , Indicates the sample number.

[0062] Step 3. For the obtained image visual features and question text features, use the attention mechanism to process the dense interactions within the image and between the image and the question text, respectively, to obtain semantically aligned image visual features and question text features.

[0063] Step 3.1. The attention mechanism includes an encoder module Encoder and a decoder module Decoder.

[0064] The encoder module consists of The self-attention units are stacked, and the self-attention unit is represented as ; The decoder module consists of The self-attention and guided attention are stacked, and the guided attention is expressed as ,like Figure 3 shown.

[0065] and After linear splicing by multiple groups of parallel multi-head attention, they are input to the feedforward layer. After the splicing and feedforward operations, there is a layer normalization operation, and finally the processed feature vector is output. Input into Encoder, and convert the image visual features and problem characteristics Enter into Deocder.

[0066] Step 3.2. Self-Attention Unit It is composed of multi-head scaled dot-product attention.

[0067] The input of scaled dot-product attention is , and , a set of image visual features input into the attention mechanism is represented as ,but The inputs are There are three different mappings of Figure 3 As shown in (a) in .

[0068] For each vector calculate .

[0069] single correspond Group key-value pairs ,Will Packaged into three matrices and ;in, Representation vector Dimension.

[0070] right Normalization is performed, that is, .

[0071] Then use Function processing Get its attention weight, and finally use the attention weight to multiply The output of scaled dot-product attention is obtained by weighted summation, which is expressed as:

[0072] .

[0073] Step 3.3. Group scaled dot-product attention to get multi-head attention, Output After processing, the features The formula is:

[0074] f = M A ( Q U , K E , V A ) = C o n c a t [ h e a d 1 , h e a d 2 , . . . , h e a d h ] W o ;

[0075] .

[0076] in, represents the normalized matrix; is the projection matrix, is a learnable parameter, represents the number of stacked attention network layers, Represents the number of attention heads.

[0077] Step 3.4. Represent a set of image visual features that are input into the attention mechanism as , the text features of the question are expressed as ; The input is also , and ,like Figure 3 As shown in (b) in .

[0078] in, In the input for The mapping of for The mapping of X = [ x 1 , . . . , x n ] , Y = [ y 1 , . . . , y n ] ; Represent the characteristics of each image, The text features of each problem representing a diseased image.

[0079] exist In, use guide To learn attention, Model the pairwise relationship between .

[0080] Step 3.5. Encoder Group The output of Encoder is represented as X ( N ) = [ x 1 N , . . . , x n N ] , for The number of layers will and Combination, by Group and Stacked into Decoder.

[0081] in, Respectively indicate that Group Features of each diseased image after processing.

[0082] The text features of the question Y = [ y 1 , . . . , y n ] With the output of Encoder X ( N ) = [ x 1 N , . . . , x n N ] Input them into the Decoder together, and the final Decoder output is represented as Y ( N ) = [ y 1 N , . . . , y n N ] .

[0083] Respectively indicate that Group and Stacked into the text features of each question after Decoder processing.

[0084] Step 3.6. Add the first The layer is represented as , whose input features are expressed as and , the output feature is expressed as and , different layers The data transfer relationship between them is expressed by the formula:

[0085] [ X ( l ) , Y ( l ) ] = M C A ( l ) ( [ X ( l − 1 ) , Y ( l − 1 ) ] ) .

[0086] That is to say layer The output of and the output of the previous layer of each Decoder are combined as the input of this layer of Decoder, and the output of the last layer of Decoder is used as the output of the entire attention mechanism model, such as Figure 4 shown.

[0087] Will layer As Encoder, use The visual features of the diseased image processed by the Encoder of the previous layer and the output features of the Decoder of the previous layer are taken as the input of each Decoder of the previous layer, and finally the output features of the Encoder and Decoder are obtained.

[0088] Step 4. Use the multimodal fusion model to deeply fuse the semantically aligned image and question text modal features obtained in step 3 to obtain the fused feature vector.

[0089] Step 4.1. Obtain image visual features through attention mechanism and question text features , use a three-dimensional matrix to fuse the features of these two modalities and obtain the fused feature ;in, The dimension representing the visual features of the image, The dimension representing the characteristics of the question text, Indicates the length of the answer text feature.

[0090] Step 4.2. Assumptions is a three-dimensional tensor, Perform Tucker decomposition and convert Decomposed into three coefficient matrices and And a core tensor , this decomposition process is expressed as follows:

[0091] ;

[0092] in, and ; Represents the dimensions of the normalized matrix. represents the dimension of question text features in the core tensor, Represents the dimension of the visual features of the diseased image in the core tensor.

[0093] You can To sum up  = [  c , W q , W v , W o ] .

[0094] Step 4.3. Represent the image visual features as , the text features of the question are expressed as , the feature fusion tensor The decomposition process is expressed as follows:

[0095] ;

[0096] The extracted core tensor The bias balancing strategy in the proposed crop disease visual question answering model based on prior knowledge is put into training. If we define , ,in , .

[0097] The complete bilinear interaction of the two modalities, image and question text, is expressed as ,Will The formula is:

[0098] ;

[0099] in .

[0100] Will Projection to prediction space , ; Then use the activation function to get , using nonlinear function processing and ,Right now , .

[0101] Step 5. Use the bias balancing strategy to process the fused feature vector to reduce the single-peak bias learned by the visual question answering model and guide the model to answer correctly according to the visual features of the disease image, such as Figure 5 shown.

[0102] An important defect of the visual question-answering task is that it has text bias, which may be reflected in the following situations in the crop disease task: for example, the color of most of the strawberry anthracnose lesions in the data set is reddish brown. Then, by learning these data, the visual question-answering model will have an inherent impression of the color of the strawberry anthracnose lesions, that is, it is reddish brown. Then, when encountering questions about the color of strawberry anthracnose lesions during the test, even if the image is other colors (such as yellow), the visual question-answering model will most likely still ignore the true features of the image and answer according to the inherent impression, that is, answer reddish brown. The crop disease visual question-answering model ILCD based on prior knowledge proposed in the present invention can analyze the true features of the disease image through a bias balance strategy, that is, use yellow to answer correctly.

[0103] Specifically, the present invention designs a deviation balancing strategy and uses question text features as masks to capture question text deviations based on conventional visual question answering models, so as to change the prediction of the crop disease visual question answering model ILCD based on prior knowledge, such as Figure 5 As shown in the figure, the answer score distribution of the conventional visual question answering model is (0.5, 0.5, 0.0), and it is impossible to get a reliable answer. The answer score distribution based on the question text features is (0.9, 0.3, 0.3). After the question text features are used as a mask to cover the answer score distribution of the visual question answering model, the final answer score distribution is (0.8, 0.3, 0.0), and a reliable answer can be obtained through this distribution. In other words, the bias balance mechanism changes the model loss by dynamically changing the answer score distribution during the model training process, and finally makes the output answer of the crop disease visual question answering model based on prior knowledge closer to the real visual features of the image.

[0104] Step 5.1. Express the deviation balance strategy as , using the text encoder Get the first Characteristics of the sample of questions , define a matrix mapping neural network And the classifier .

[0105] represents the answer text features, represents the mask dimension, Represents the dimension of the question text.

[0106] The deviation balance strategy is expressed as:

[0107] .

[0108] Step 5.2. Dynamically delete the parameters of the bias balancing strategy during ILCD training.

[0109] use Function processing get ,right Take the square to get the mask , then the mask Perform a dot product operation with the output of ILCD, that is, use the mask to modify the prediction of the visual question answering model to dynamically change the loss.

[0110] The visual question answering model with bias balancing strategy is defined as , using the formula It is expressed as:

[0111] ;

[0112] in represents the output of the complete bilinear interaction of the two modalities of image and question text, express function.

[0113] Then use As a classifier , processing the overall multimodal fusion features covered by the mask, a set of answer text The answer is the one with the highest score.

[0114] Step 6. Train a classifier to classify the low unimodal deviation features processed in step 5 to obtain the visual question answering results of crop diseases.

[0115] Step 6.1. The crop disease visual question answering model based on informed learning uses two losses to calculate the gradient to jointly optimize the model parameters; among them, the loss It is a crop disease visual question answering model based on prior knowledge The prediction is generated; BCELoss is used to calculate the loss of the main model, using Indicates all caused The model parameters.

[0116] Step 6.2. Loss Produced by the bias balance strategy, it is related to the question text features The Cross-Entropy Loss associated with the prediction of Indicates all caused The model parameters.

[0117] The input features of the bias balancing strategy are the same as the output of the question text encoder of the visual question answering model; in order to prevent Directly learn the problem bias, and set the loss of the bias balancing strategy so that it will not be back-propagated to .

[0118] Step 6.3. Combine the visual question answering model and the bias balance strategy as the main model ILCD. Add the two losses in step 6.1 and step 6.2 to get the loss of the final model. , back-propagate this loss; The formula used is: .

[0119] At this point, a crop disease visual question-answering model based on prior knowledge is obtained.

[0120] For the input disease images, disease images with disease spots were taken in farmland environments, and data enhancement was performed using methods such as rotation, cropping, and adding Gaussian noise. When training the model, the image with the target box marked was used as the prior knowledge disease image data, and the entire image was used as the regular disease image data; when testing the model, the entire image was input to the model as the input of the model's visual features. For the input question texts, agricultural experts designed question texts that can reflect the visual features of the disease, such as question texts about color, size, and location. After data enhancement using methods such as keyword replacement and sentence changes, these designed question texts were used as the question texts for the input model. The question texts input during model training and testing are the same.

[0121] The trained crop disease visual question-answering model inputs crop disease images and question texts, uses pre-trained image and question text feature extractors to extract features of the two modal data, then uses a common attention mechanism to process intensive interactions between the two modalities and within each modality, and then uses a multimodal feature fusion model to fuse the feature vectors of the two modalities processed by the common attention mechanism, and then uses the deviation balancing strategy proposed in the present invention to reduce the unimodal deviation, so that the model answers according to the real content of the image, improves the prediction accuracy of the model, and finally uses a classifier to obtain the output answer.

[0122] In order to prove the feasibility and superiority of the model trained by the method of the present invention, a comparative test was conducted between the model of the present invention (ILCD) and the prior art. The models involved in the comparative experiment are as follows:

[0123] The Stacked Attention Network (SAN) uses a stacked attention mechanism to handle visual question answering tasks, gradually focusing on the most relevant parts of the image and question; Grounded Visual Question Answering (GVQA) associates natural language questions with specific areas in the image and answers them based on the question and image information; Language-Visual Multimodal BERT (LXMERT) is a joint learning model that is trained on tasks such as visual question answering, natural language reasoning, and image understanding to obtain multimodal representations; Reducing Unimodal Biases in Visual Question Answering (RUBi) reduces the model's bias towards unimodal data through self-supervised learning and adversarial training, thereby improving the generalization ability of multimodal tasks; Gradient Reversal Layer model (GRL) is used for domain adaptation and domain adversarial training to reduce the distribution difference between the source domain and the target domain during training; Hierarchical Neuron Concept Explainer model (Hierarchical Neuron Concept Explainer The HINT model gradually introduces different levels of importance by training the network in layers to improve the generalization performance of the model; the end-to-end context-aware detection method (ECD) is an end-to-end context-aware detection method used to improve the performance and efficiency of object detection tasks; the attribute-oriented mechanisms and Memory networks model (AVM) combines memory networks and attribute-oriented mechanisms to achieve more flexible and accurate visual reasoning.

[0124] The experimental results are shown in Table 1.

[0125] Table 1 Performance comparison of various models

[0126]

[0127] Among them, All, Yes / No, Number, and Other respectively represent the dataset category as all datasets, questions asking whether it is correct, questions asking counts, and other questions. The larger the values ​​of these evaluation indicators, the stronger the performance of the model in the visual question answering task. Through experiments, it can be seen that compared with the models SAN and GVQA with simple stacked attention mechanisms, ILCD leads by more than 20% in accuracy on both datasets. Compared with the adaptive model GRL and the distributed network HINT, ILCD leads by 7.42% and 3.02% respectively, and the training time and model parameter amount of ILCD are maintained at a low level. ILCD achieved the best results in the values ​​of All, Yes / No, Number, and Other on the two public datasets, VQA-CP v2 and VQA v2.

[0128] The crop disease visual question-answering method proposed in the present invention can effectively solve the problem that the existing visual question-answering model cannot obtain multi-attribute visual information of crop disease images and judge the stage of disease occurrence according to this information. The method of the present invention effectively solves the problem that the single-peak deviation misleads the model training, resulting in the model being unable to produce a true description according to the crop disease images, and improves the model's prediction accuracy for disease images in different conditions.

[0129] Example 2

[0130] This embodiment 2 describes a crop disease visual question answering system, which is based on the same inventive concept as the crop disease visual question answering method described in the above embodiment 1.

[0131] A crop disease visual question-answering system, comprising the following modules:

[0132] The preprocessing module is used to collect, preprocess and label crop disease visual question-answering data;

[0133] The crop disease prior knowledge VQA data is mixed into the conventional crop disease VQA data set to obtain the crop disease visual question answering data set containing prior knowledge of the present invention. The method of training a model using such a data set is called informed learning;

[0134] A feature extraction module is used to extract the visual features and question text features of each image using an image visual feature extractor and a text feature extractor, and then input them into a crop disease visual question answering model based on prior knowledge;

[0135] The attention module is used to process the dense interactions within the image and between the image and the question text using the attention mechanism to obtain the image visual features and question text features, and obtain the semantically aligned image visual features and question text features;

[0136] The multimodal fusion module is used to obtain the fused feature vector by deeply fusing the semantically aligned image and question text features obtained by the multimodal fusion model;

[0137] The bias balance module is used to process the fused feature vector using the bias balance strategy to reduce the single-peak bias learned by the visual question answering model and force the model to answer correctly according to the visual features of the disease image;

[0138] and a classification module, which is used to train a classifier to classify the low unimodal deviation features obtained after the deviation balancing strategy processing to obtain the visual question-answering results of crop diseases.

[0139] It should be noted that in the crop disease visual question-answering system of this embodiment 2, the implementation process of the functions and effects of each functional module is specifically described in the implementation process of the corresponding steps in the method of the above embodiment 1, and will not be repeated here.

[0140] Example 3

[0141] This embodiment 3 describes a computer device, which includes a memory and one or more processors. The memory stores executable code. When the processor executes the executable code, it is used to implement the steps of the crop disease visual question-answering method in the above embodiment 1.

[0142] In this embodiment, the computer device is any device or apparatus with data processing capability, which will not be described in detail here.

[0143] Example 4

[0144] This embodiment 4 describes a computer-readable storage medium on which a program is stored. When the program is executed by a processor, it is used to implement the steps of the crop disease visual question-answering method in the above embodiment 1.

[0145] The computer-readable storage medium may be an internal storage unit of any device or apparatus with data processing capabilities, such as a hard disk or memory, or an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc., equipped on the device.

[0146] Of course, the above description is only a preferred embodiment of the present invention, and the present invention is not limited to the above embodiments. It should be noted that all equivalent substitutions and obvious deformation forms made by any technician familiar with the field under the guidance of this specification fall within the essential scope of this specification and should be protected by the present invention.

Claims

1. A crop disease visual question answering method, characterized in that: The steps include: Step 1. Collect, preprocess and label crop disease visual question answering data; The crop disease prior knowledge VQA data is mixed into the conventional crop disease VQA data set to obtain a crop disease visual question answering data set containing prior knowledge. The method of training the model using this data set is called informed learning. Step 2. Build a crop disease visual question answering model based on prior knowledge, which includes an image visual feature extractor, a question text feature extractor, an attention mechanism, a multimodal fusion model, a bias balance strategy, and a classifier; Firstly, the image visual feature extractor and question text feature extractor are used to extract the visual features and question text features of each image in the crop disease visual question answering dataset containing prior knowledge. Step 3. Based on the obtained image visual features and question text features, the attention mechanism is used to process the dense interactions within the image and between the image and the question text, respectively, to obtain semantically aligned image visual features and question text features; Step 4. Use the multimodal fusion model to deeply fuse the semantically aligned image visual features and question text features obtained in step 3 to obtain a fused feature vector; Step 5. Use the bias balancing strategy to process the fused feature vector to reduce the single-peak bias learned by the crop disease visual question answering model and guide the model to answer correctly according to the visual features of the disease image; Step 6. Train a classifier to classify the low unimodal deviation features obtained after the deviation balance strategy in step 5, and finally obtain the visual question answering results of crop diseases; The step 2 is specifically as follows: Step 2.

1. The input of the visual question answering model is divided into two modal data: crop disease images and disease-related question texts. The image modal data is represented as V, where the i-th image is represented as v i ; The question text modal data is represented as Q, and the i-th question text of a disease image is represented as q i , the annotated answer text is represented as A,q i The corresponding answer text is represented as a i ; The overall crop disease dataset is represented as Contains multiple v i -q i -a i right; Step 2.

2. Normalize each image input to the model to a preset size; in order to handle variable question length m, use zero padding to make each question vector equal in size; Step 2.

3. Given a pre-trained image encoder e v (·) and the text encoder e q (·); Using e v (·) and e q (·) Encode the crop disease image and question text respectively to obtain the features of the corresponding modality, where the visual features of the image are represented by e v (v i ), the question text feature is represented as e q (q i ), i represents the sample number; The step 3 is specifically as follows: Step 3.

1. The attention mechanism includes an encoder module Encoder and a decoder module Decoder; The encoder module is composed of L layers of self-attention units stacked together, and the self-attention unit is denoted as SA; The decoder module is composed of L stacked sets of self-attention and guided attention, where guided attention is represented by GA; the visual features of the image e v (v i ) is input into the Encoder, and the visual features of the image are v (v i ) and question text features e q (q i ) Input into Deocder; Step 3.

2. The self-attention unit SA is composed of multi-head scaled dot-product attention; The input of scaled dot-product attention is qu:query, ke:key and va:value. A set of image visual features input into the attention mechanism is represented as X. Then the input of SA is three different mappings of X respectively. For each vector Calculate score = qu × ke; single Corresponding to n sets of key-value pairs Encapsulate qu, ke, and va into three matrices respectively and Among them, d ke represents the dimension of the vector ke; Normalize the score, that is Then use the softmax function to process the score to obtain its attention weight, and finally use the attention weight to multiply VA for weighted summation to get the output of scaled dot-product attention, which is expressed as: Step 3.

3. Concatenate h groups of scaled dot-product attention to get multi-head attention, and output the features processed by SA from the multi-head attention f. The formula of f is: f=MA(QU,KE,VA)=Concat[head1,head2,...,head h ]W o ; Among them, W o represents the normalized matrix, is the projection matrix, is a learnable parameter, d represents the number of stacked attention network layers, and d h Indicates the number of attention heads; Step 3.

4. Denote a set of image visual features that are input into the attention mechanism as X, and the text features of the question as Y; the input of GA is also qu:query, ke:key and va:value; Among them, qu in the input of GA is the mapping of X, ke,va is the mapping of Y; X=[x1,...,x n ],Y=[y1,...,y n ]; x1,...,x n Represent the features of each image, y1,...,y n The text features representing each problem of a diseased image; Use Y to guide X to learn attention in GA and model the pairwise relationship between <X,Y>; Step 3.

5. Encoder is composed of N groups of SA stacked together. The output of Encoder is expressed as N is the number of SA layers. SA and GA are combined, and N groups of SA and GA are stacked to form a Decoder. in, They represent the characteristics of each disease image after N groups of SA processing; The text feature of the question Y = [y1,...,y n ] and the output of Encoder Input them into the Decoder together, and the final Decoder output is represented as Represents the text features of each question after N groups of SA and GA are stacked into Decoder processing; Step 3.

6. Represent the lth layer in Encoder-Decoder as MCA (l) , whose input feature is represented by X (l-1) and Y (l-1) , the output feature is represented as X (l) and Y (l) , the data transfer relationship between MCAs of different layers is expressed by the formula: [X (l) ,Y (l) ]=MCA (l) ([X (l-1) ,Y (l-1) ]); The step 4 is specifically as follows: Step 4.

1. Obtain image visual features X through attention mechanism (L) and question text features Y (L) , use a three-dimensional matrix to fuse the features of these two modalities and obtain the fused feature Among them, d q Represents the dimension of the image visual features, d v The dimension representing the characteristics of the question text, Indicates the length of the answer text; Step 4.

2. Assumptions is a three-dimensional tensor, Perform Tucker decomposition and convert Decomposed into three coefficient matrices W q , W v and W o And a core tensor This decomposition process is expressed as follows: in, and t o represents the dimension of the normalized matrix; t q Represents the dimension of question text features in the core tensor, t v Represents the dimension of the visual features of the disease image in the core tensor; Step 4.

3. Represent the image visual features as v Τ W v , the text feature of the question is represented as q Τ W q , the feature fusion tensor The decomposition process is expressed as follows: If you define in The complete bilinear interaction of the two modalities of image and question text is expressed as z, and z is expressed as: in Project z to the prediction space y=z Τ W o , Then use the activation function to get p = softmax (y), and use the nonlinear function to process and Right now The step 5 is specifically as follows: Step 5.

1. Design a bias balancing strategy. Based on the obtained fusion features, use the question text features as a mask to cover the multimodal fusion features of the traditional visual question answering model to capture the question text deviation. The question text features used as masks are the same as the question text features of the input model to dynamically change the multimodal fusion features and subsequent prediction results. Denote the deviation balance strategy as f Q : Using text encoder q (·) Get the feature e of the i-th problem sample q (q i ), define a matrix mapping neural network rn q : and classifier c q : in Represents the answer text feature, n q represents the mask dimension, d q Represents the dimension of the question text; The deviation balance strategy is expressed as: f Q (q i )=c q (rn q (e q (q i ))); Step 5.

2. Dynamically delete the parameters of the bias balance strategy when training the crop disease visual question answering model based on prior knowledge; Use sigmoid function to process rn q Get rm, square rm to get mask rm 2 , then mask rm 2 Perform a dot multiplication with the output of the crop disease visual question answering model based on prior knowledge, that is, use the mask to modify the prediction of the visual question answering model to dynamically change the loss; The visual question answering model with bias balancing strategy is defined as f QM , use the formula to convert f QM It is expressed as: f QM =z⊙σ(rn q (e q (q i ))); Where z represents the complete bilinear interaction between the two modalities of image and question text, and σ represents the sigmoid function; Then use softmax as the classifier c q , processing the overall multimodal fusion features covered by the mask, a set of answer text The highest scoring answer; The step 6 is specifically as follows: Step 6.

1. The crop disease visual question answering model based on prior knowledge uses two losses to calculate the gradient to jointly optimize the model parameters; among them, the loss It is a crop disease visual question answering model based on prior knowledge f QM The prediction is generated; BCELoss is used to calculate the loss of the main model, using θ QM Indicates all caused Model parameters of Step 6.

2. Loss Produced by the bias balance strategy, it is related to the question text feature f Q (q i ) prediction-related Cross-Entropy Loss, using θ QO Indicates all caused Model parameters of The input features of the bias balancing strategy are the same as the output of the question text encoder of the visual question answering model; in order to prevent e q Directly learn the problem bias, and set the loss of the bias balancing strategy will not be back-propagated to e q ; Step 6.

3. Add the two losses in step 6.1 and step 6.2 to get the loss of the final model Backpropagate this loss; The formula used is: in, represents the overall crop disease dataset; The step 1 is specifically as follows: Step 1.

1. Searching for original disease image data, the original disease image data contains multiple categories; Select high-quality image data with high pixels, clear lesion edges and internal visual features from the original disease image data, and then perform data enhancement on the selected high-quality image data; Next, select selected images of equal proportion for each disease from the enhanced images, where the selected images have clear imaging, standard fruit and lesion features, and complete targets; Label each selected image with several annotation boxes, mark the important objects in the image, and obtain the image data of VQA data of prior knowledge of crop diseases; Finally, the image data of the crop disease prior knowledge VQA data labeled with the conventional disease image data are mixed for the training of the crop disease visual question answering model, and the labeled images are added as informed data to the informed learning; Step 1.

2. Design a series of visual attributes of different image focus angles, including disease type, crop type, shape, color, state, texture, quantity, location, and size of lesions and fruits; For the image data in the conventional crop disease VQA dataset, a variety of questions are raised based on the above visual attributes. These questions are evenly assigned to each crop disease image as its question text annotation data; For images of informed learning tasks, we design question texts for each feature in the annotation box one by one, and present its visual features to the model to the greatest extent; then we perform data augmentation on the question texts; Finally, we annotate each disease image or image annotation box with several questions focusing on different visual attributes, and construct image-question text and image annotation box-question text pair data; Step 1.

3. Label the answer for each question; each question is labeled with several answers with different confidence levels. Among these answers, the items with high confidence and large number are selected as the correct answers and provided to the model for learning.

2. A crop disease visual question answering system for implementing the crop disease visual question answering method according to claim 1, characterized in that: The crop disease visual question answering system includes the following modules: A preprocessing module is used to collect, preprocess and annotate crop disease visual question answering data, and mix the crop disease prior knowledge visual question answering data set into the conventional crop disease visual question answering data set to obtain the crop disease visual question answering data set containing prior knowledge of the present invention. The method of using such a data set to train a model is called informed learning; A feature extraction module is used to first extract the visual features and question text features of each image using a feature extractor before inputting the obtained image into the visual question answering model, and then input them into the crop disease visual question answering model respectively; The attention module is used to process the dense interactions within the image and between the image and the question text respectively based on the obtained image visual features and question text features, and obtain the image visual features and question text features; The multimodal fusion module is used to obtain the fused feature vector by deeply fusing the semantically aligned image and question text features obtained by the multimodal fusion model; The bias balance module is used to process the fused feature vector using the bias balance strategy to reduce the single-peak bias learned by the visual question answering model and guide the model to answer correctly according to the visual features of the disease image; And a classification module is used to train a classifier to classify the processed low unimodal deviation features to obtain the visual question-answering results of crop diseases.

3. A computer device comprising a memory and one or more processors; an executable code is stored in the memory; characterized in that: When the processor executes the executable code, it is used to implement the crop disease visual question-answering method described in claim 1 above.

4. A computer-readable storage medium having a program stored thereon; characterized in that: When the program is executed by a processor, it is used to implement the crop disease visual question-answering method described in claim 1.

Images