A generalized zero-shot pest and disease image classification method based on causal gating model

Abstract

The invention discloses a generalized zero-shot pest image classification method based on a causal gating model, which belongs to the field of image data processing technology, and specifically includes the following steps: collecting original pest images of known and unknown classes to construct a data set, and dividing it into a training set and a test set; constructing a causal feature extraction module, and designing a loss function, and optimizing the causal feature extraction module based on the training set and the loss function; constructing a two-stream hypersphere alignment module, and designing a loss function, and optimizing the two-stream hypersphere alignment module based on the training set and the loss function; using the trained two-stream hypersphere alignment module to assign a binary domain label to each pest image in the test set, and then obtaining the final causal gating model; using a camera to directly shoot the pest image to be classified in the current farmland, and inputting the final causal gating model to obtain the pest image classification result. The invention can accurately identify pests and diseases of unknown categories.

Description

Technical Field

[0001] The present invention belongs to the technical field of image data processing, and in particular relates to a generalized zero-sample pest and disease image classification method based on a causal gating model. Background Art

[0002] Pests and diseases can cause huge economic losses to crops. In order to ensure agricultural productivity and quality, it is crucial to accurately identify pests and diseases and use pesticides to eliminate them. However, some pest and disease categories have only a small number of image instances, or even no image instances, so it is impossible to form a training set to train the pest and disease image classification model in a supervised learning manner. In this case, the model will misclassify unknown pests and diseases, or even fail to identify them at all.

[0003] In order to solve this problem, the prior art provides a technical solution: using a generative zero-shot learning solution to classify unknown pests and diseases, that is, training a generative adversarial network to generate visual features for unknown pests and diseases, and then using the generated visual features to train a classifier to complete the image classification task of the unknown pests and diseases category is a solution. However, the method based on generative zero-shot learning has an inherent problem, that is, the distribution of the generated unknown class visual features is different from the distribution of the real unknown class visual features, resulting in the classifier trained with the former having poor performance on conventional zero-shot image classification tasks. This problem is called the feature confusion problem. This problem will cause the model to have low accuracy when identifying unknown categories of pests and diseases. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes a generalized zero-shot pest image classification method based on a causal gating model, that is, firstly, a binary classifier is trained to determine whether a pest image is from a known domain or an unknown domain, and then a classifier based on supervised learning training or a classifier based on conventional zero-shot learning training is used to process samples from known domains and unknown domains respectively. This method decomposes the generalized zero-shot learning task into conventional zero-shot learning tasks and supervised learning tasks, thereby solving the problem of feature confusion and improving the recognition accuracy of unknown pests and diseases.

[0005] The technical solution of the present invention is as follows:

[0006] A generalized zero-shot pest and disease image classification method based on a causal gating model, the causal gating model includes a causal feature extraction module and a two-stream hypersphere alignment module, and the classification method specifically includes the following steps:

[0007] Step 1: Collect original images of pests and diseases of known and unknown categories to construct a data set, and divide it into a training set and a test set;

[0008] Step 2: Build a causal feature extraction module, design a loss function, and optimize the causal feature extraction module based on the training set and the loss function;

[0009] Step 3: construct a dual-stream hypersphere alignment module, design a loss function, and optimize the dual-stream hypersphere alignment module based on the training set and the loss function;

[0010] Step 4: Use the trained two-stream hypersphere alignment module to assign a binary domain label to each pest image in the test set, and then obtain the final causal gating model;

[0011] Step 5: Use a camera to directly capture the pest and disease images to be classified in the current farmland, and input the final causal gating model to obtain the pest and disease image classification results.

[0012] Furthermore, in step 1, the original images of pests and diseases are collected by agricultural practitioners using cameras from multiple crops in multiple farmlands, all the collected original images of pests and diseases are constructed into a data set, the data set is input into a pre-trained ResNet101 network to obtain a visual feature set, and the visual feature set is divided into a training set and a test set in proportion;

[0013] The intermediate features obtained after each sample in the training set is input into the ResNet101 network are used as the required visual features x. All visual features x constitute the visual feature set. x(i) is the i-th visual feature, corresponding to the i-th sample; n is the number of samples in the training set; N is the dimension.

[0014] Furthermore, in step 2, the causal feature extraction module includes two encoders and one decoder, and is optimized by three different losses; the specific working process and loss function of the causal feature extraction module are as follows:

[0015] Step 2.1: Input the visual features x into the first encoder f c and the second encoder f aug , using mutual information loss Decouple the visual feature x into two independent features. The output of the first encoder is the causal feature x. c =f c (x), the output of the second encoder is the enhanced feature x′ aug =f aug (x);

[0016] Step 2.2: Use the contrast log ratio to estimate the upper bound of the mutual information value of two variables. The formula is:

[0017]

[0018] in, is the mutual information estimation function of the upper bound of the contrast logarithmic ratio; i and j are the numbers of two different samples; For a given conditions, The conditional probability density function of ; and Enhance the i-th sample and j-th sample in the feature set respectively; is the causal feature set The i-th sample of , the i-th sample also corresponds to the i-th causal feature; For a given conditions, The logarithmic ratio of the conditional probability density function;

[0019] Step 2.3: Assume that the training set is passed through the first encoder to obtain the causal feature set After the second encoder, the enhanced feature set is obtained To enhance the i-th sample in the feature set; first, aug The index of x′ is randomly rearranged and x′ aug and x c Add element by element. The calculation process is expressed as follows:

[0020]

[0021] in, is the enhanced feature set after random shuffling; Shuffle(·) means random shuffling;

[0022] Step 2.4: and Transpose and redefine as the Transposed Augmented Feature Set Transpose the causal feature set in for The i-th sample in ; express The i-th sample in corresponds to the set of all i-th dimension elements of causal features in a batch; based on this, maximize the and The correlation is used to approximate the common cause principle, and the optimization objective is expressed as:

[0023]

[0024] Where COR(·) represents the function for calculating the correlation coefficient between two variables; g c For the decoder;

[0025] Step 2.5: Minimize the number of and The correlation between the causal factors is used to make them independent of each other, and the optimization objective formula is expressed as:

[0026]

[0027] in, for The jth sample in ;

[0028] Step 2.6: Construct a correlation matrix to simultaneously calculate the two optimization objectives in step 1.5 and step 1.6. The correlation matrix is ​​expressed as follows:

[0029]

[0030] Among them, C i，j For a batch and The correlation matrix of ; <·, ·> represents the inner product operation; ||·||2 is the l2 norm;

[0031] Step 2.7. Loss function It is expressed by the formula:

[0032]

[0033] Among them, ||·|| F represents the Frobenius norm; C is the correlation matrix; I represents the identity matrix;

[0034] Step 2.8: Introduce reconstruction loss To use x′ aug and x c Reconstruct x and obtain the reconstructed visual features It is expressed by the formula:

[0035]

[0036] Here, ||·||1 represents the l1 norm.

[0037] Furthermore, in step 3, the two-stream hypersphere alignment module includes a two-stream autoencoder of a visual stream and an attribute stream, wherein the visual stream includes a visual encoder f v and a visual decoder g v , the attribute stream contains an attribute encoder f a and an attribute decoder g a , and is optimized by four different losses; the working process and loss function of the two-stream hypersphere alignment module are as follows:

[0038] Step 3.1: The input of the two-stream hypersphere alignment module is the causal feature xc and class-level attributes a, which are large amounts of text about pest and disease category-related knowledge collected from Baidu Encyclopedia and Wikipedia, and mapped into vectors using the Doc2Vec tool;

[0039] Step 3.2: Design visual reconstruction loss The formula is:

[0040]

[0041] Among them, ||·||1 is the l1 norm; Reconstructing causal feature processes for visual flow, Reconstructing causal feature processes for attribute flows,

[0042] Step 3.3: Design attribute reconstruction loss The formula is:

[0043]

[0044] in, Refactoring of class-level attribute procedures for attribute flows, Reconstructing class-level attribute processes for visual flow,

[0045] Step 3.4: Given a causal feature set The label set {y1,...,y i , ..., y n}、Class-level attribute set corresponding to the causal feature set y i is the causal feature set The i-th causal feature of The corresponding i-th label, is the causal feature set The i-th causal feature of The corresponding i-th class-level attribute; in the latent space, the design center loss The calculation formula is:

[0046]

[0047] Among them, exp(·) is an exponential function with the natural constant e as the base; is the i-th causal feature mapped to the latent space, is the i-th class-level attribute mapped to the latent space, d represents the dimension of the latent vector; is the i-th causal feature mapped into the latent space; K is the number of known pest and disease categories; is the class level of the kth known pests and diseases; S is the category of known pests and diseases;

[0048] Step 3.5: Introduce the angle linear layer to project the latent vector onto the hypersphere for binary domain detection. The angle linear layer is expressed by the formula:

[0049]

[0050] Among them, x h Represents x′ c Projection vector on the unit hypersphere; x′ c is the causal feature x′ mapped to the latent space c ; is a hyperparameter; m is the angular margin; θ represents W T and x′ c The angle between them has a value range of [0, π]; W represents the weight of the fully connected layer, and T is the transposed symbol;

[0051] Similarly, mapping a′ to the unit hypersphere is expressed as:

[0052]

[0053] Among them, a h represents the projection vector of a′ on the unit hypersphere; a′ is the class-level attribute mapped into the latent space;

[0054] Step 3.6: When x h and a h After the calculation is completed, the A-Sonmax loss is introduced to complete the classification task of known pests and diseases, which is expressed by the formula:

[0055]

[0056] in, is the A-Softmax loss value; express Projection vector on the unit hypersphere; express and The angle between is the transpose of the weights of the fully connected layer of the i-th label; express and The angle between For the The transpose of the weights of the fully connected layer for each label; is the number of the label set; n is the number of samples in the training set; ψ(·) is a monotonically decreasing function;

[0057] Step 3.7: Train the dual-stream hypersphere alignment module by using the gradient descent method and the back propagation method to obtain a trained dual-stream hypersphere alignment module.

[0058] Furthermore, the specific process of step 4 is as follows:

[0059] Step 4.1: Set the causal feature set. and attribute sets Project into spherical space, and then calculate each visual feature in spherical space and its corresponding class-level attributes The cosine similarity of the same category is collected into a set Finally, search for a value lower than The cosine similarity γ of most values ​​in , which can ensure The ratio of the cosine similarity greater than γ to the total cosine similarity is equal to the manually set threshold η; the process is expressed by the formula:

[0060]

[0061] Among them, c is the cosine similarity set Samples of

[0062] Step 4.2: Given the cosine similarity γ, complete the binary domain detection task. The formula is:

[0063]

[0064] Among them, y d Represents the binary domain label of the test sample, 0 means it belongs to an unknown class domain, and 1 means it belongs to a known class domain; Represents the visual features x in the test set te Projection vector in spherical space; is the projection vector of the class-level attributes of the known pest and disease categories in the latent space;

[0065] Step 4.3: After the binary domain labels of the test samples are determined, for the known class test samples, directly use the The loss and the two-stream hypersphere alignment module trained in step 3.7 are used to classify the known class test samples; for the unknown class test samples, the existing f-CLSWGAN method is used to synthesize the visual features of the unknown class pest and disease images. The process is expressed as:

[0066]

[0067] in, represents the visual features of the synthesized unknown pest and disease images; f_CLSWGAN(·) is the f-CLSWGAN method; Class-level attributes representing unknown pest and disease categories;

[0068] Step 4.4: Use After a two-layer fully connected layer module and a softmax function, the probability distribution of each unknown class is obtained. This process is expressed as follows:

[0069]

[0070] Among them, θ cls Represents the fully connected layer module MLP cls (·) parameters; represents the probability distribution of each unknown class; Softmax(·) is the softmax function;

[0071] Step 4.5: Use cross entropy loss to train the fully connected layer module proposed in step 4.4 to classify unknown pests and diseases. The formula of cross entropy loss is expressed as:

[0072]

[0073] Among them, L CE represents the cross entropy loss; y u Represents the true probability distribution of unknown pest and disease categories.

[0074] Furthermore, the specific process of step 5 is: inputting the pest image to be classified into the pre-trained ResNet101 network to obtain visual features, and then inputting the visual features into the trained causal gating model to identify the binary domain label of the current pest image; according to the binary domain label category, classifying the pest image through the classification method described in steps 4.3 to 4.5.

[0075] The beneficial technical effects brought by the present invention are as follows: The present invention proposes a generalized zero-shot pest and disease image classification method based on a causal gating model, which can solve the problem that the existing pest and disease image classification lacks a binary domain classification scheme for known pest and disease domains and unknown pest and disease domains. The method effectively solves the feature confusion problem in the generalized zero-shot pest and disease image classification task and can accurately identify pests and diseases of unknown categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 The flowchart of the generalized zero-shot pest and disease image classification method based on the causal gating model of the present invention.

[0077] Figure 2The present invention is a flow chart of extracting causal features of pest and disease images using a causal feature extraction module.

[0078] Figure 3 The present invention is a flow chart of aligning causal features and class-level attributes on a hypersphere using a dual-stream hypersphere alignment module. DETAILED DESCRIPTION

[0079] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0080] The present invention realizes a generalized zero-shot pest image classification method based on a causal gating model. The causal gating model consists of a causal feature extraction module and a two-stream hypersphere alignment module. The causal feature extraction module takes the original visual features of the pest image as input, and then outputs the causal features of the pest image. The causal features and the class-level attributes of the pest are used as inputs of the two-stream hypersphere alignment module and aligned in the latent space. Based on this method, the problem that the existing pest image classification lacks a binary domain classification scheme for known pest domains and unknown pest domains can be solved. This method effectively solves the feature confusion problem in the generalized zero-shot pest image classification task and improves the recognition accuracy of unknown pests.

[0081] like Figure 1 As shown, the method of the present invention mainly comprises the following steps:

[0082] Step 1: Collect original images of pests and diseases of known and unknown categories to construct a data set, and divide it into a training set and a test set;

[0083] The original images of pests and diseases are collected by agricultural practitioners using cameras from multiple crops in multiple fields. All the collected original images of pests and diseases are constructed into a dataset, which is input into the pre-trained ResNet101 network to obtain a visual feature set, and the visual feature set is divided into a training set and a test set in proportion.

[0084] The ResNet101 network is a prior art and is a deep residual network network. The core of the network is to let each residual block learn the difference between input and output, rather than directly learning the mapping relationship. Specifically, after the input of each residual block passes through some convolutional layers, the result is added to the original input and then output. In this way, each residual block can learn additional information beyond the identity mapping, so that the network can learn more complex functions. The present invention pre-trains the ResNet101 network on the ImageNet dataset; the ResNet101 network used in the present invention removes the last classifier, and the intermediate features obtained after each sample in the training set is input into the ResNet101 network are used as the required visual feature x, and all visual features x constitute a visual feature set. x(i) is the i-th visual feature, i.e., the i-th sample; n is the number of samples in the training set, is the visual feature set in the training set; N is the dimension;

[0085] Step 2: construct a causal feature extraction module, design a loss function, and optimize the causal feature extraction module based on the training set and the loss function; the causal feature extraction module is used to extract the causal features of the pest and disease images;

[0086] The causal feature extraction module contains two encoders and one decoder, and is optimized by three different losses.

[0087] like Figure 2 As shown in Figure 2, the specific working process and loss function of the causal feature extraction module are as follows:

[0088] Step 2.1: Input the visual features x into the first encoder f c and the second encoder f aug , using mutual information loss Decouple the visual feature x into two independent features. The output of the first encoder is the causal feature x. c =f c (x), the output of the second encoder is the enhanced feature x′ aug =f aug (x);

[0089] Step 2.2: Since x′ aug and x c The joint probability density function of is unknown, so x′ cannot be directly calculated aug and x c In order to approximate the mutual information value of two variables, the upper bound of the mutual information value of the two variables is estimated using the contrast logarithm ratio. The calculation process can be expressed as follows:

[0090]

[0091] Among them, I CLUB (·) is the mutual information estimation function of the contrast logarithm ratio upper bound; is log2p(x′ aug |x c ) is the mathematical expectation value; p(x c , x′ aug ) is x′ aug and x c The joint probability density function of c ) and p(x′ aug ) are x c and x′ aug The marginal probability density function of p(x′aug |x c ) is a given x c Under the condition, x′ aug The conditional probability density function of , which is usually approximated by a deep neural network.

[0092] By minimizing I CLUB The causal feature extraction module is optimized in an upper bound manner so that x′ aug and x c Independent of each other, the above I CLUB is the theoretical function for estimating the mutual information value. In actual operation, I CLUB Available Equivalent replacement, the specific calculation process can be expressed by the formula:

[0093]

[0094] in, is the mutual information estimation function of the upper bound of the contrast logarithmic ratio; i and j are the numbers of two different samples; For a given conditions, The conditional probability density function of ; and Enhance the i-th sample and j-th sample in the feature set respectively; is the causal feature set The i-th sample of , the i-th sample is the i-th causal feature; For a given conditions, The logarithm ratio of the conditional probability density function of .

[0095] Step 2.3: Introducing the “self-enhancement” operation and factorization loss To force x c Satisfy the common cause principle and independent causal mechanism properties of the causal gating model. Specifically, assume that the training set is obtained after the first encoder to obtain the causal feature set After the second encoder, the enhanced feature set is obtained is the i-th sample in the enhanced feature set. First, aug The index of x′ is randomly rearranged and x′ aug and x c Add element by element. The calculation process can be expressed as:

[0096]

[0097] in, is the enhanced feature set after random shuffling; Shuffle(·) indicates random shuffling.

[0098] Step 2.4, the “self-enhancement” operation makes the second encoder f aug tends to extract noise in visual features, while making the first encoder f c Tends to extract non-noise features from the original visual features. After completing the above operations, and Transpose and redefine as the Transposed Augmented Feature Set Transpose the causal feature set in for The i-th sample in ; express The i-th sample in corresponds to the set of all i-th dimension elements of causal features in a batch. Based on this, we can maximize and The correlation of is used to approximate the common cause principle. The optimization objective can be expressed as:

[0099]

[0100] Where COR(·) represents the function for calculating the correlation coefficient between two variables; g c For the decoder.

[0101] Step 2.5: The independent causal mechanism property stipulates that the causal factors in the causal feature are independent of each other, so it can minimize the number of causal factors in a batch. and The correlation between the causal factors is used to make them independent of each other. The optimization objective can be expressed as:

[0102]

[0103] in, for The jth sample in ;

[0104] Step 2.6: For ease of calculation, a correlation matrix is ​​constructed to simultaneously calculate the two optimization objectives in step 1.5 and step 1.6. The correlation matrix can be expressed as:

[0105]

[0106] Among them, C i，j For a batch and The correlation matrix of ; <·, ·> represents the inner product operation; ||·||2 is the l2 norm, that is, the Euclidean distance.

[0107] The optimization objectives of step 2.7, step 2.4, and step 2.5 can be equivalent to optimizing the diagonal values ​​of the correlation matrix to 1 to meet the requirements of the common cause principle, and optimizing the off-diagonal values ​​to 0 to meet the requirements of independent causal mechanisms. Therefore, the loss function It can be expressed by the formula:

[0108]

[0109] Among them, ||·|| F represents the Frobenius norm; C is the correlation matrix; I represents the identity matrix.

[0110] Step 2.8, based on and The causal features and non-causal features can be made independent of each other, but the causal feature extraction module expects that the causal features and non-causal features contain all the information of the original visual features, so the reconstruction loss is introduced To use x′ aug and x c Reconstruct x and obtain the reconstructed visual features It can be expressed by the formula:

[0111]

[0112] Here, ||·||1 represents the l1 norm.

[0113] Step 3: construct a two-stream hypersphere alignment module, design a loss function, and optimize the two-stream hypersphere alignment module based on the training set and the loss function; the two-stream hypersphere alignment module is used to align the causal features and class-level attributes of the pest and disease images on the hypersphere;

[0114] The two-stream hypersphere alignment module contains a two-stream autoencoder of the visual stream and the attribute stream, where the visual stream contains a visual encoder f v and a visual decoder g v , the attribute stream contains an attribute encoder f a and an attribute decoder g a , and is optimized by four different losses. Figure 3 As shown in Figure 2, the working process and loss function of the dual-stream hypersphere alignment module are as follows:

[0115] Step 3.1: The input of the two-stream hypersphere alignment module is the causal feature x c and class-level attribute a, which is a large amount of text about pest and disease category-related knowledge collected from Baidu Encyclopedia and Wikipedia and mapped into vectors using the Doc2Vec tool, where K is the number of known pest and disease categories, is the attribute set, S is the known pest category, k is the number of the known pest category, is the class-level attribute of the kth known pest.

[0116] Step 3.2: On the one hand, the two-stream hypersphere alignment module reconstructs causal features from the visual stream. This process can be expressed as On the other hand, in order to align the representations of causal features and class-level attributes in the latent space, the two-stream hypersphere alignment module reconstructs the causal features from the attribute stream. This process can be expressed as Based on this, the l1 norm distance between the input causal features and the reconstructed causal features is measured as the visual reconstruction loss therefore It can be expressed by the formula:

[0117]

[0118] Among them, ||·||1 is the l1 norm; Reconstructing causal feature processes for visual flow; Reconstructing causal feature processes for attribute flows;

[0119] Step 3.3: On the one hand, the two-stream hypersphere alignment module reconstructs the class-level attributes from the attribute stream. This process can be expressed as On the other hand, in order to align the representations of causal features and class-level attributes in the latent space, the two-stream hypersphere alignment module reconstructs the class-level attributes from the visual stream. The process can be expressed as Based on this, the l1 norm distance between the input class-level attributes and the reconstructed class-level attributes is measured as the attribute reconstruction loss therefore It can be expressed by the formula:

[0120]

[0121] in, Refactoring of class-level attribute procedures for attribute flows; Reconstructing class-level attribute processes for visual flow;

[0122] Step 3.4: In order to improve the class information richness of the latent vector, use the center loss Make the distances between vectors belonging to different classes in the latent space farther, and make the distances between vectors belonging to the same class closer. Specifically, given a causal feature set The label set {y1,...,y i , ..., y n}、Class-level attribute set corresponding to the causal feature set y i is the causal feature set The i-th causal feature of The corresponding i-th label, is the causal feature set The i-th causal feature of The corresponding i-th class-level attribute; first, the causal features and class-level attributes are mapped to the latent space, and the process is expressed as and Where d represents the dimension of the latent vector; is the i-th causal feature mapped into the latent space; is the i-th class-level attribute mapped to the latent space. In the latent space, The calculation process can be expressed by the formula:

[0123]

[0124] Where exp(·) is an exponential function with the natural constant e as the base; k is the number of the known pest category; K is the number of the known pest categories; is the class level of the kth known pests and diseases; S is the category of known pests and diseases;

[0125] Step 3.5: Introduce the angle linear layer to project the latent vector onto the hypersphere for binary domain detection. and Omit superscript (i) and subscript y i , represented as the causal feature x′ mapped to the latent space c and mapped to the class-level attribute a′ in the latent space, assuming x′ c and a′ pass through a fully connected layer, denoted as W T x′ c +b and W T a′+b, where W and b represent the weight and bias of the fully connected layer respectively, and T is the transposed symbol. c has the same dimension as a′ and has been aligned in the latent space, so here we only use x′ c Take W as an example to introduce the subsequent steps. T x′ c +b is rewritten as ||W T ||||x′ c ||cos(θ)+b, where θ represents W T and x′ c The angle between them is in the range [0, π]. Then force W T Normalize and set b to 0, expressed as ||W T|1=1,b=0, this operation causes the vectors in the latent space to be distributed only in an angular distribution, that is, distributed on the unit hypersphere. In order to make the decision boundary stricter, an integer m is introduced to provide an angular margin. Specifically, cos(θ) is replaced by a monotonically decreasing function is a hyperparameter. Therefore, the angle linear layer can be expressed by the formula:

[0126]

[0127] Among them, x h Represents x′ c Projection vector onto the unit hypersphere.

[0128] Similarly, mapping a′ to the unit hypersphere is expressed as:

[0129]

[0130] Among them, a h Represents the projection vector of a′ on the unit hypersphere.

[0131] Step 3.6: When x h and a h After the calculation is completed, the A-Softmax loss is introduced to complete the classification task of known pests and diseases categories, which can be expressed by the formula:

[0132]

[0133] in, is the A-Softmax loss value; express Projection vector on the unit hypersphere; express and The angle between is the transpose of the weights of the fully connected layer of the i-th label; express and The angle between For the The transpose of the weights of the fully connected layer for each label; is the number of the label set; n is the number of samples in the training set; ψ(·) is a monotonically decreasing function. Figure 3 In the figure, the dotted circle represents the latent space, x′ c Projection vector x on the unit hypersphere h is the element type, the projection vector a of a′ on the unit hypersphere h is the center of mass.

[0134] Step 3.7: Train the dual-stream hypersphere alignment module by using the gradient descent method and the back propagation method to obtain a trained dual-stream hypersphere alignment module.

[0135] Step 4: Use the trained two-stream hypersphere alignment module to assign a binary domain label to each pest image in the test set, and then obtain the final causal gating model. This process includes the following steps:

[0136] Step 4.1: Set the above causal feature set and attribute sets Project into spherical space, and then calculate each visual feature in spherical space and its corresponding class-level attributes The cosine similarity of the same category is collected into a set Finally, search for a value lower than The cosine similarity γ of most values ​​in , which can ensure The ratio of cosine similarity greater than γ to all cosine similarities is equal to the manually set threshold η. The process can be expressed by the formula:

[0137]

[0138] Among them, c is the cosine similarity set Samples of

[0139] Step 4.2: Given the cosine similarity γ, the binary domain detection task can be completed, that is, to determine whether each test sample in the test set belongs to a known class domain or an unknown class domain. The judgment method is expressed by the formula:

[0140]

[0141] Among them, y d Represents the binary domain label of the test sample, 0 means it belongs to an unknown class domain, and 1 means it belongs to a known class domain; Represents the visual features x in the test set te Projection vector in spherical space; is the projection vector of the class-level attributes of the known pest and disease categories in the latent space.

[0142] Step 4.3: After the binary domain labels of the test samples are determined, for the known class test samples, directly use the The loss and step 3.7 training complete the two-stream hypersphere alignment module to classify the known class test samples. For the unknown class test samples, the existing f-CLSWGAN method is used to synthesize the visual features of the unknown class pest and disease images. The process can be expressed as:

[0143]

[0144] in, represents the visual features of the synthesized unknown pest and disease images; f_CLSWGAN(·) is the f-CLSWGAN method; A class-level attribute representing the unknown pest category.

[0145] Step 4.4: Use After a two-layer fully connected layer module and a softmax function, the probability distribution of each unknown class is obtained. This process can be expressed as:

[0146]

[0147] Among them, θ cls Represents the fully connected layer module MLP cls (·) parameters; represents the probability distribution of each unknown class; Softmax(·) is the softmax function.

[0148] Step 4.5: Use the cross entropy loss to train the fully connected layer module proposed in step 4.4 to achieve the classification of unknown pests and diseases. The formula of cross entropy loss can be expressed as:

[0149]

[0150] Among them, L CE represents the cross entropy loss; y u Represents the true probability distribution of unknown pest and disease categories.

[0151] Step 5: Use a camera to directly shoot the pest image to be classified in the current farmland, and input it into the final causal gating model to obtain the classification result of the pest image. The specific process is: input the pest image to be classified into the pre-trained ResNet101 network to obtain visual features, and then input the visual features into the trained causal gating model to identify the binary domain label of the current pest image. According to the binary domain label category, the pest image is classified by the classification method described in steps 4.3 to 4.5.

[0152] In order to demonstrate the feasibility and superiority of the model trained by the method of the present invention, the following comparative experiments are given.

[0153] The method of the present invention is compared with ten existing methods, including f-CLSWGAN, cycle-CLSWGAN, GAZSL, ZSML, AMAZ, MGA-GAN, COSMO, OOD, GatingAE, and DUS-VAE, on two data sets: pest set and disease set. Among them, the f-CLSWGAN method improves the quality and diversity of generated images by learning category-specific boundaries; the cycle-CLSWGAN method is used to convert images between different domains and maintain the consistency of category features; the GAZSL method aims to classify categories that have never been seen by linking visual features and semantic feature spaces; the ZSML method achieves classification of unknown categories by learning the mapping relationship between visual features and semantic features; the AMAZ method uses the self-attention mechanism to capture long-distance dependencies in images to generate more realistic images; the MGA-GAN method is used to generate consistent images from a variety of different modal data; the COSMO method generates images with specified attributes by learning category-specific boundaries; the OOD method identifies and excludes abnormal images by learning the feature distribution of normal images; the GatingAE method introduces a gating mechanism to extract important features in images and achieve image reconstruction; the DUS-VAE method is a variational autoencoder used to learn the potential distribution of data and achieve data generation and reconstruction. The comparative experimental results are shown in Table 1:

[0154] Table 1 Performance comparison of each method;

[0155]

[0156] Among them, U is the accuracy of unknown categories, S is the accuracy of known categories, and H is the harmonic mean accuracy. The larger the values ​​of U, S and H, the stronger the performance of the model in the generalized zero-shot learning image classification task. It can be seen from the experimental results that the method of the present invention has achieved the best prediction results, and the values ​​of U, S and H are all maximum values.

[0157] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A generalized zero-shot pest image classification method based on a causal gating model, characterized in that: The causal gating model includes a causal feature extraction module and a two-stream hypersphere alignment module. The classification method specifically includes the following steps: Step 1: Collect original images of pests and diseases of known and unknown categories to construct a data set, and divide it into a training set and a test set; Step 2: Build a causal feature extraction module, design a loss function, and optimize the causal feature extraction module based on the training set and the loss function; In step 2, the causal feature extraction module includes two encoders and one decoder, and is optimized by three different losses; the specific working process and loss function of the causal feature extraction module are as follows: Step 2.1: Input the visual features x into the first encoder f c and the second encoder f aug , using mutual information loss Decouple the visual feature x into two independent features. The output of the first encoder is the causal feature x. c =f c (x), the output of the second encoder is the enhanced feature x′ aug =f aug (x); Step 2.2: Use the contrast log ratio to estimate the upper bound of the mutual information value of two variables. The formula is: in, is the mutual information estimation function of the upper bound of the contrast logarithmic ratio; i and j are the numbers of two different samples; For a given conditions, The conditional probability density function of ; and Enhance the i-th sample and j-th sample in the feature set respectively; is the causal feature set The i-th sample of , the i-th sample also corresponds to the i-th causal feature; For a given conditions, The logarithmic ratio of the conditional probability density function; Step 2.3: Assume that the training set is passed through the first encoder to obtain the causal feature set After the second encoder, the enhanced feature set is obtained To enhance the i-th sample in the feature set; first, aug The index of x′ is randomly rearranged and x′ aug and x c Add element by element, and the calculation process is expressed as follows: in, is the enhanced feature set after random shuffling; Shuffle(·) means random shuffling; Step 2.4: and Transpose and redefine as Transpose Augmented Feature Set Transpose the causal feature set in for The i-th sample in ; express The i-th sample in corresponds to the set of all i-th dimension elements of causal features in a batch; based on this, maximize the and The correlation is used to approximate the common cause principle, and the optimization objective is expressed as: Where COR(·) represents the function for calculating the correlation coefficient between two variables; g c For the decoder; Step 2.5: Minimize the number of and The correlation between the causal factors is used to make them independent of each other, and the optimization objective formula is expressed as: in, for The jth sample in ; Step 2.6: Construct a correlation matrix to simultaneously calculate the two optimization objectives in step 1.5 and step 1.

6. The correlation matrix is ​​expressed as follows: Among them, C i,j For a batch and The correlation matrix of ; <·,·> represents the inner product operation; ||·||2 is the l2 norm; Step 2.

7. Loss function It is expressed by the formula: Among them, ||·|| F represents the Frobenius norm; C is the correlation matrix; I represents the identity matrix; Step 2.8: Introduce reconstruction loss To use x′ aug and x c Reconstruct x and obtain the reconstructed visual features It is expressed by the formula: Among them, ||·||1 represents the l1 norm; Step 3: construct a dual-stream hypersphere alignment module, design a loss function, and optimize the dual-stream hypersphere alignment module based on the training set and the loss function; Step 4: Use the trained two-stream hypersphere alignment module to assign a binary domain label to each pest image in the test set, and then obtain the final causal gating model; Step 5: Use a camera to directly capture the pest and disease images to be classified in the current farmland, and input the final causal gating model to obtain the pest and disease image classification results.

2. The generalized zero-shot pest and disease image classification method based on the causal gating model according to claim 1 is characterized in that: In the step 1, the original images of pests and diseases are collected by agricultural practitioners using cameras for multiple crops in multiple farmlands, all the collected original images of pests and diseases are constructed into a data set, the data set is input into a pre-trained ResNet101 network to obtain a visual feature set, and the visual feature set is divided into a training set and a test set in proportion; The intermediate features obtained after each sample in the training set is input into the ResNet101 network are used as the required visual features x. All visual features x constitute the visual feature set. x (i) is the i-th visual feature, corresponding to the i-th sample; n is the number of samples in the training set; N is the dimension.

3. The generalized zero-shot pest and disease image classification method based on the causal gating model according to claim 1, characterized in that: In step 3, the two-stream hypersphere alignment module includes a two-stream autoencoder of a visual stream and an attribute stream, wherein the visual stream includes a visual encoder f v and a visual decoder g v , the attribute stream contains an attribute encoder f a and an attribute decoder g a , and is optimized by four different losses; the working process and loss function of the two-stream hypersphere alignment module are as follows: Step 3.1: The input of the two-stream hypersphere alignment module is the causal feature x c and class-level attributes a, which are large amounts of text about pest and disease category-related knowledge collected from Baidu Encyclopedia and Wikipedia, and mapped into vectors using the Doc2Vec tool; Step 3.2: Design visual reconstruction loss The formula is: Among them, ||·||1 is the l1 norm; Reconstructing causal feature processes for visual flow, Reconstructing causal feature processes for attribute flows, Step 3.3: Design attribute reconstruction loss The formula is: in, Refactoring of class-level attribute procedures for attribute flows, Reconstructing class-level attribute processes for visual flow, Step 3.4: Given a causal feature set The label set {y1,…,y i ,…,y n }、Class-level attribute set corresponding to the causal feature set y i is the causal feature set The i-th causal feature of The corresponding i-th label, is the causal feature set The i-th causal feature of The corresponding i-th class-level attribute; in the latent space, the design center loss The calculation formula is: Among them, exp(·) is an exponential function with the natural constant e as the base; is the i-th causal feature mapped to the latent space, is the i-th class-level attribute mapped to the latent space, d represents the dimension of the latent vector; is the i-th causal feature mapped into the latent space; K is the number of known pest and disease categories; is the class level of the kth known pests and diseases; S is the category of known pests and diseases; Step 3.5: Introduce the angle linear layer to project the latent vector onto the hypersphere for binary domain detection. The angle linear layer is expressed by the formula: Among them, x h Represents x c ′ is the projection vector on the unit hypersphere; x′ c is the causal feature x mapped to the latent space c ′; is a hyperparameter; m is the angular margin; θ represents W T and x′ c The angle between them has a value range of [0,π]; W represents the weight of the fully connected layer, and T is the transposed symbol; Similarly, mapping a′ to the unit hypersphere is expressed as: Among them, a h represents the projection vector of a′ on the unit hypersphere; a′ is the class-level attribute mapped into the latent space; Step 3.6: When x h and a h After the calculation is completed, the A-Softmax loss is introduced to complete the classification task of known pests and diseases categories, which is expressed by the formula: in, is the A-Softmax loss value; express Projection vector on the unit hypersphere; express and The angle between is the transpose of the weights of the fully connected layer of the i-th label; express and between The angle of For the The transpose of the weights of the fully connected layer for each label; is the number of the label set; n is the number of samples in the training set; ψ(·) is a monotonically decreasing function; Step 3.7: Train the dual-stream hypersphere alignment module by using the gradient descent method and the back propagation method to obtain a trained dual-stream hypersphere alignment module.

4. The generalized zero-shot pest and disease image classification method based on the causal gating model according to claim 3 is characterized in that: The specific process of step 4 is as follows: Step 4.1: Set the causal feature set and attribute sets Project into spherical space, and then calculate each visual feature in spherical space and its corresponding class-level attributes The cosine similarity of the same category is collected into a set Finally, search for a value lower than The cosine similarity γ of most values ​​in , which can ensure The ratio of cosine similarity greater than γ to all cosine similarities is equal to the manually set threshold η; The process is expressed by the formula: Among them, c is the cosine similarity set Samples of Step 4.2: Given the cosine similarity γ, complete the binary domain detection task. The formula is: Among them, y d Represents the binary domain label of the test sample, 0 means it belongs to an unknown class domain, and 1 means it belongs to a known class domain; Represents the visual features x in the test set te Projection vector in spherical space; is the projection vector of the class-level attributes of the known pest and disease categories in the latent space; Step 4.3: After the binary domain labels of the test samples are determined, for the known class test samples, directly use the The loss and the two-stream hypersphere alignment module trained in step 3.7 are used to classify the known class test samples; for the unknown class test samples, the existing f-CLSWGAN method is used to synthesize the visual features of the unknown class pest and disease images. The process is expressed as: in, represents the visual features of the synthesized unknown pest and disease images; f_CLSWGAN(·) is the f-CLSWGAN method; Class-level attributes representing unknown pest and disease categories; Step 4.4: Use After a two-layer fully connected layer module and a softmax function, the probability distribution of each unknown class is obtained. This process is expressed as follows: Among them, θ cls Represents the fully connected layer module MLP cls (·) parameters; represents the probability distribution of each unknown class; Softmax(·) is the softmax function; Step 4.5: Use cross entropy loss to train the fully connected layer module proposed in step 4.4 to classify unknown pests and diseases. The formula of cross entropy loss is expressed as: Among them, L CE represents the cross entropy loss; y u Represents the true probability distribution of unknown pest and disease categories.

5. The generalized zero-shot pest and disease image classification method based on the causal gating model according to claim 4 is characterized in that: The specific process of step 5 is: input the pest image to be classified into the pre-trained ResNet101 network to obtain visual features, and then input the visual features into the trained causal gating model to identify the binary domain label of the current pest image; according to the binary domain label category, classify the pest image through the classification method of steps 4.3 to 4.5.

Images