A small sample pest image recognition method based on causal intervention technology

Abstract

The present invention relates to a small sample pest image recognition method based on causal intervention technology, which solves the defect of low recognition rate of small sample pest images compared with the prior art. The present invention comprises the following steps: acquisition of public data sets; construction and training of embedding models; acquisition, labeling and preprocessing of small sample pest images; construction and training of small sample pest image recognition models. The present invention utilizes causal intervention to stratify features, alleviates the influence of confounding factors in pre-training knowledge, improves the accuracy of pest image recognition in small sample conditions, and increases the robustness of small sample pest image recognition algorithms.

Description

Technical Field

[0001] The invention relates to the field of small sample pest image recognition, and in particular to a small sample pest image recognition method based on causal intervention technology. Background Art

[0002] In recent years, with the continuous development of artificial intelligence technology, deep learning has been widely used in scenarios such as face recognition, autonomous driving, and speech recognition, and has achieved great breakthroughs. However, in the context of deep learning, a large amount of labeled data is often required to enable deep learning models to be applied. In real life, humans are different. Compared with machines, humans are better at recognizing new things through a small number of samples. Therefore, people very much hope that machines can have the ability to associate and memorize and learn quickly from a small number of samples, and hope that artificial intelligence will be closer to human intelligence.

[0003] At the same time, in agricultural scenarios, it is sometimes very expensive, difficult, or even impossible to collect and annotate a large amount of data on a certain type of pest. Therefore, the research topic of small-sample learning of pest image recognition has great application value.

[0004] In machine learning, the model mainly focuses on the correlation between features, but it is difficult to identify the causality between features. Therefore, how to introduce causal intervention to scientifically identify the causal relationship between features and greatly reduce the number of samples required for unseen image recognition has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The purpose of the present invention is to solve the defect of low recognition rate of small sample pest images in the prior art, and to provide a small sample pest image recognition method based on causal intervention technology to solve the above problem.

[0006] In order to achieve the above object, the technical solution of the present invention is as follows:

[0007] A small sample pest image recognition method based on causal intervention technology includes the following steps:

[0008] Acquisition of public datasets: Obtain a large number of datasets from public datasets and divide them into base datasets and validation sets for training and selecting embedding models respectively;

[0009] Build and train the embedding model: Use the base class dataset to train the embedding model, and use the validation set during the training process to select the appropriate embedding model for feature extraction;

[0010] Acquisition, labeling and preprocessing of small sample pest images: Acquire a small sample image dataset, label and preprocess the small sample pest images, where the labeled small sample pest image data is used as the support set data, and the unlabeled small sample pest images are used as the query set data;

[0011] Build and train a small sample pest image recognition model: Use the trained embedding model to build a small sample pest image recognition model, and use the support set and query set data to train it, and obtain the small sample pest image recognition results.

[0012] The construction and training of the embedding model includes the following steps:

[0013] Building an embedding model;

[0014] ResNet-12 is used as the embedding model, which contains four residual blocks, and the number of output channels is 64, 160, 320 and 640 respectively;

[0015] Each residual block is set to contain three convolution operations, all of which use a 3×3 convolution kernel, and a normalization operation is used after each convolution operation;

[0016] It is set that each residual block is followed by maximum pooling, and the fourth residual block is also subjected to global average pooling, and the features output by the average pooling are used as the embedded features of the image;

[0017] It is set that during pre-training, according to the number of categories of the base class data, a multi-classification linear classifier is connected to predict the base class category, and a four-classification linear classifier is connected to predict the angle of image rotation; during fine-tuning, according to the number of categories of the small sample pest image, a multi-classification linear classifier is connected to fine-tune and predict the category of the small sample pest;

[0018] Train the embedding model;

[0019] Use rotational self-supervision to pre-train embedding models;

[0020] In the process of training the embedding model, an additional four-category linear classifier is used to predict the rotation angle of the image, and the rotation self-supervised loss function L rotation for:

[0021] L roation =L(C4(f θ (x r )),r),

[0022] Among them, r is the rotation angle, x is the input image data, and x r is a picture rotated by r degrees, L is the standard cross entropy loss, f θ (x r) is the output of the embedding model, C4 is a four-category classifier, r∈{0°, 90°, 180°, 270°};

[0023] Use manifold mixture to pre-train embedding models;

[0024] When training each batch of images, a network layer of the embedding model is randomly selected, and the feature maps of these data on this network layer are mixed manifolds;

[0025] Characteristics of manifold mixing and tags They are:

[0026]

[0027]

[0028] The loss function after manifold mixing is:

[0029]

[0030] Among them, f l (x1) and f l (x2) are the training data (x i ,y i ) and (x j ,y j ) is the feature representation of the lth layer in the embedding model; β∈[0,1]; y i and j All are one-hot labels; L is the standard cross entropy loss; g is the model representation after layer l;

[0031] For each batch of image data, all images are first rotated by 0°, 90°, 180°, and 270°, and then manifold blending is performed;

[0032] For manifold blending, manifold blending is performed not only on the unrotated image, but also on the rotated image, which means that the label of the rotation angle is also transformed in the manifold blending;

[0033] After each cycle of data training is completed, multiple small sample tasks are sampled using the validation set data. Similarly, each small sample task contains a small amount of support set data and a large amount of query set data. Next, the embedding model is fixed, and the support set data in the small sample task on the validation set is used to fine-tune the classifier of the last layer. Finally, the query set data in the small sample task is tested using the fine-tuned model, and the embedding model with the best test result is saved to obtain a trained embedding model.

[0034] After training an embedding model, reset the random seed used during training and train a new embedding model, thereby training n embedding models.

[0035] The construction and training of a small sample pest image recognition model comprises the following steps:

[0036] Feature extraction operation: Use the trained embedding model to extract features from the support set and query set data, and perform feature processing on the extracted features;

[0037] Use transductive learning or inductive learning to process the features obtained above;

[0038] The resulting features are used to fine-tune multiple classifiers and tested on unlabeled query set images.

[0039] The feature extraction operation includes the following steps:

[0040] In this small sample pest identification task, each support set data is first randomly cropped k times and input into the embedding model;

[0041] After averaging the k output feature vectors, the resulting feature vector is used as the feature of the support set data.

[0042] Since n different embedding models are used, n different feature vectors corresponding to the same data are obtained.

[0043] Concatenate these n vectors as the features of the small sample image And perform a center transformation on this feature:

[0044]

[0045]

[0046] in, It represents the feature vector obtained after feature preprocessing of a base class data, which has not undergone center transformation; represents the average feature of all base class data, which represents the feature center after the data set is mapped to the embedding space; k represents the number of base class data; x c Represents the feature vector after translating the feature origin to the feature center.

[0047] The processing of the features obtained above by using transductive learning or inductive learning includes the following steps:

[0048] If the inductive learning method is used, the previously obtained feature vector x is directly output c ;

[0049] If transductive learning is used, soft K-means is used to perform transductive operations on features;

[0050] Initialize the feature centers for each class:

[0051]

[0052] S i represents the support data, x represents the features of the support data after the above steps, represents the initial feature center of each class;

[0053] Update the center of each category by iteratively:

[0054]

[0055]

[0056]

[0057] in, represents the initial feature center of the i-th class, i represents the i-th class, t represents the t-th iteration, x′ represents the feature vector assigned to the i-th class at this time, C ind Indicates the subscript of the class center to which each query set data is assigned, Set i It represents the set of feature vectors assigned to the query set to the i-th category. β is a hyperparameter, which is 5 here. w(.) is the weight calculation function. The support set data weight is 1, and the rest are given weights less than 1 according to the distance. exp represents the e exponential function. and Represent the feature centers at time t and t+1 respectively;

[0058] Will and It is output as the final support set data to the next stage.

[0059] The method of fine-tuning multiple classifiers using the obtained features and testing the unlabeled query set pest images comprises the following steps:

[0060] During fine-tuning, each feature (x c or ) is sliced, and the features are divided into c parts for training multiple cosine classifiers. Each time, c-1 parts are taken to train a cosine classifier, and a total of c cosine classifiers are trained;

[0061] During testing, the features of each query set data are sliced ​​in the same way as during fine-tuning;

[0062] Each part of the features is input into the corresponding classifier for prediction, and the prediction results of the c classifiers are merged to obtain the small sample pest image recognition result.

[0063] Beneficial Effects

[0064] The small sample pest image recognition method based on causal intervention technology of the present invention utilizes causal intervention to stratify features compared with the prior art, alleviates the influence of confounding factors in pre-training knowledge, improves the accuracy of pest image recognition in small sample conditions, and increases the robustness of the small sample pest image recognition algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a method sequence diagram of the present invention. DETAILED DESCRIPTION

[0066] In order to have a further understanding and recognition of the structural features and the effects achieved by the present invention, a preferred embodiment and accompanying drawings are used for detailed description as follows:

[0067] like Figure 1 As shown, the small sample pest image recognition method based on causal intervention technology described in the present invention comprises the following steps:

[0068] The first step is to obtain public datasets: obtain a large number of datasets from public datasets and divide them into base datasets and validation sets for training and selecting embedding models respectively. Due to the small amount of data in the case of small samples, overfitting is prone to occur. The large number of base datasets obtained from public datasets can help the model train better and more general feature representations. At the same time, the validation set is used to test the embedding model in the training process to select a better embedding model.

[0069] The second step is to build and train the embedding model: use the base class dataset to train the embedding model, and use the validation set to select the appropriate embedding model for feature extraction during the training process. In order to train the model to have better and more general feature representations, rotational self-supervision is added during the training process. Manifold hybrid training can make the decision boundary smoother and the embedded features smoother and more uniform, thereby improving the generalization ability of the model.

[0070] The specific steps are as follows:

[0071] (1) Building an embedding model;

[0072] A1) ResNet-12 is used as the embedding model, which contains four residual blocks, and the number of output channels is 64, 160, 320 and 640 respectively;

[0073] A2) Each residual block is set to contain three convolution operations, all of which use a 3×3 convolution kernel, and a normalization operation is used after each convolution operation;

[0074] A3) Set each residual block to perform maximum pooling, and the fourth residual block also performs global average pooling, and the features output by the average pooling are used as the embedded features of the image;

[0075] A4) is set to connect a multi-class linear classifier to predict the base class category according to the number of categories of the base class data during pre-training, and connect a four-class linear classifier to predict the angle of image rotation; during fine-tuning, connect a multi-class linear classifier to fine-tune and predict the category of the small sample pest image according to the number of categories of the small sample pest image.

[0076] (2) Train the embedding model;

[0077] B1) Use rotation self-supervision to pre-train the embedding model;

[0078] In the process of training the embedding model, an additional four-category linear classifier is used to predict the rotation angle of the image, and the rotation self-supervised loss function L rotation for:

[0079] L roation =L(C4(f θ (x r )),r),

[0080] Among them, r is the rotation angle, x is the input image data, and x r is a picture rotated by r degrees, L is the standard cross entropy loss, f θ (x r ) is the output of the embedding model, C4 is a four-category classifier, r∈{0°, 90°, 180°, 270°};

[0081] B2) Using manifold mixture pre-trained embedding model;

[0082] When training each batch of images, a network layer of the embedding model is randomly selected, and the feature maps of these data on this network layer are mixed manifolds;

[0083] Characteristics of manifold mixing and tags They are:

[0084]

[0085]

[0086] The loss function after manifold mixing is:

[0087]

[0088] Among them, f l (x1) and f l (x2) are the training data (x i ,y i ) and (x j ,y j ) is the feature representation of the lth layer in the embedding model; β∈[0,1]; y i and j All are one-hot labels; L is the standard cross entropy loss; g is the model representation after layer l;

[0089] B3) For each batch of image data, first rotate all images by 0°, 90°, 180°, and 270°, and then perform manifold blending;

[0090] For manifold blending, manifold blending is performed not only on the unrotated image, but also on the rotated image, which means that the label of the rotation angle is also transformed in the manifold blending;

[0091] After each cycle of data training is completed, multiple small sample tasks are sampled using the validation set data. Similarly, each small sample task contains a small amount of support set data and a large amount of query set data. Next, the embedding model is fixed, and the support set data in the small sample task on the validation set is used to fine-tune the classifier of the last layer. Finally, the query set data in the small sample task is tested using the fine-tuned model, and the embedding model with the best test result is saved to obtain a trained embedding model.

[0092] After training an embedding model, the random seed used during training is reset to train a new embedding model, and n embedding models are trained in this way. Here, different embedding models can be trained by setting different random seeds, and the features extracted by different embedding models are different. Therefore, using multiple embedding models can make the extracted features complementary.

[0093] The third step is to obtain, label and preprocess small sample pest images: obtain a small sample image dataset, label and preprocess the small sample pest images, where the labeled small sample pest image data is used as the support set data, and the unlabeled small sample pest images are used as the query set data.

[0094] The fourth step is to build and train a small sample pest image recognition model: Use the trained embedding model to build a small sample pest image recognition model, and use the support set and query set data to train it, and obtain the small sample pest image recognition results.

[0095] Since the embedding model extracts common features, and in a small sample data set, the amount of data is small, which leads to the extraction and attention of some unimportant features, such as background, etc. These features are confounding factors that affect the model judgment. The purpose of causal intervention is to alleviate and eliminate the impact of these confounding factors.

[0096] Here, the causal intervention method used can be summarized as follows: in the process of pre-training the embedding model, different random number seeds are set to train n different embedding models. Then, the n features of the same image after inputting into the n embedding models are concatenated and processed as the features of the data. On this basis, each time this feature is sampled And at the same time train m cosine classifiers to predict category Y:

[0097]

[0098] in, is the concatenation operation, P is the probability, do is the do operator, c is the sampling operation, and the sampling of the original features The use of this causal intervention method can alleviate the impact of confounding factors in the original embedded model and effectively improve the accuracy of small sample classification.

[0099] The specific steps are as follows:

[0100] (1) Feature extraction operation: Use the trained embedding model to extract features from the support set and query set data, and perform feature processing on the extracted features.

[0101] C1) In this small sample pest identification task, each support set data is randomly cropped k times and input into the embedding model;

[0102] After averaging the k output feature vectors, the resulting feature vector is used as the feature of the support set data.

[0103] Since n different embedding models are used, n different feature vectors corresponding to the same data are obtained.

[0104] C2) concatenate these n vectors as the features of the small sample image And perform a center transformation on this feature:

[0105]

[0106]

[0107] in, It represents the feature vector obtained after feature preprocessing of a base class data, which has not undergone center transformation; represents the average feature of all base class data, which represents the feature center after the data set is mapped to the embedding space; k represents the number of base class data. In practice, k is taken as 30, the purpose of which is to minimize the deviation of the original image features; x c Represents the feature vector after translating the feature origin to the feature center.

[0108] (2) Use transductive learning or inductive learning to process the features obtained above.

[0109] D1) If the inductive learning method is used, the previously obtained feature vector x is directly output c ;

[0110] D2) If transductive learning is used, soft K-means is used to perform transductive operations on features;

[0111] D21) Initialize the feature center of each class:

[0112]

[0113] S i represents the support data, x represents the features of the support data after the above steps, represents the initial feature center of each class;

[0114] D22) Update the center of each category by iteration:

[0115]

[0116]

[0117]

[0118] in, represents the initial feature center of the i-th class, i represents the i-th class, t represents the t-th iteration, x′ represents the feature vector assigned to the i-th class at this time, C ind Indicates the subscript of the class center to which each query set data is assigned, Set i It represents the set of feature vectors assigned to the query set to the i-th category. β is a hyperparameter, which is 5 here. w(.) is the weight calculation function. The support set data weight is 1, and the rest are given weights less than 1 according to the distance. exp represents the e exponential function. and Represent the feature centers at time t and t+1 respectively;

[0119] D23) and It is output as the final support set data to the next stage.

[0120] (3) Use the obtained features to fine-tune multiple classifiers and test them on unlabeled query set images.

[0121] E1) During fine-tuning, each feature (x c or ) is sliced, and the features are divided into c parts for training multiple cosine classifiers. Each time, c-1 parts are taken to train a cosine classifier, and a total of c cosine classifiers are trained; during testing, the features of each query set data are sliced, and the slicing method is consistent with that during training; finally, each part of the features is input into the corresponding classifier for prediction, and the prediction results of these c classifiers are merged. In practice, c is 5.

[0122] E2) During testing, the features of each query set data are sliced ​​in the same way as during fine-tuning;

[0123] E3) Input each part of the features into the corresponding classifier for prediction, and merge the prediction results of the c classifiers to obtain the small sample pest image recognition result.

[0124] Table 1 Comparison of experimental results of this scheme and other schemes, 5 categories (unit, %)

[0125]

[0126] Table 1 shows the comparison of this scheme and other small-shot methods on three public datasets: mini-ImageNet, CUB, and CIFAR-FS. The small-shot task is set to 5 categories, with 1 sample data per class labeled (1-Shot) or 5 sample data per class labeled (5-Shot). It can be seen that when using transductive learning, our method is optimal when using 3 embedding models. On the 5-category small-shot task of the mini-ImageNet dataset, the accuracy of 1 or 5 labeled samples per class reached 78.12% and 88.72% respectively; on the 5-category small-shot task of the CUB dataset, the accuracy of 1 or 5 labeled samples per class reached 86.10% and 93.35% respectively; on the 5-category small-shot task of the CIFAR-FS dataset, the accuracy of 1 or 5 labeled samples per class reached 83.58% and 90.72% respectively. The data after "±" in the table are accuracy fluctuations.

[0127] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A small sample pest image recognition method based on causal intervention technology, characterized in that: The following steps are involved: 11) Acquisition of public datasets: A large number of datasets are obtained from public datasets and divided into base datasets and validation sets for training and selecting embedding models respectively; 12) Build and train the embedding model: Use the base class dataset to train the embedding model, and use the validation set to select the appropriate embedding model for feature extraction during the training process; 13) Acquisition, labeling and preprocessing of small sample pest images: Acquire a small sample image dataset, label and preprocess the small sample pest images, wherein the labeled small sample pest image data is used as the support set data, and the unlabeled small sample pest images are used as the query set data; 14) Construct and train a small sample pest image recognition model: Use the trained embedding model to construct a small sample pest image recognition model, and use the support set and query set data to train it, and obtain the small sample pest image recognition results; The construction and training of a small sample pest image recognition model comprises the following steps: 141) Feature extraction operation: Use the trained embedding model to extract features from the support set and query set data, and perform feature processing on the extracted features; The feature extraction operation includes the following steps: 41) In the small sample pest identification task, each support set data is randomly cropped k times and then input into the embedding model; After averaging the k output feature vectors, the resulting feature vector is used as the feature of the support set data. Since n different embedding models are used, n different feature vectors corresponding to the same data are obtained. 42) Concatenate these n vectors as features of the small sample image And perform a center transformation on this feature: in, It represents the feature vector obtained after feature preprocessing of a base class data, which has not undergone center transformation; represents the average feature of all base class data, which represents the feature center after the data set is mapped to the embedding space; k represents the number of base class data; x c Represents the feature vector after translating the feature origin to the feature center; 142) Using transductive learning or inductive learning to process the features obtained above; The processing of the features obtained above by using transductive learning or inductive learning includes the following steps: 51) If the inductive learning method is used, the previously obtained feature vector x is directly output c ; 52) If transductive learning is used, soft K-means is used to perform transductive operations on features; 521) Initialize the feature center of each class: S i represents the support data, x represents the features of the support data after the above steps, represents the initial feature center of each class; 522) Update the center of each category by iteration: in, represents the initial feature center of the i-th class, i represents the i-th class, t represents the t-th iteration, x′ represents the feature vector assigned to the i-th class at this time, C ind Indicates the subscript of the class center to which each query set data is assigned, Set i It represents the set of feature vectors assigned to the query set to the i-th category. β is a hyperparameter, which is 5 here. w(.) is the weight calculation function. The support set data weight is 1, and the rest are given weights less than 1 according to the distance. exp represents the e exponential function. and Represent the feature centers at time t and t+1 respectively; 523) and It is output as the final support set data to the next stage; 143) Use the obtained features to fine-tune multiple classifiers and test them on unlabeled query set images; The method of fine-tuning multiple classifiers using the obtained features and testing the unlabeled query set pest images comprises the following steps: 61) During fine-tuning, first perform fine-tuning on each feature xc or Slice the features and divide them into c parts for training multiple cosine classifiers. Take c-1 parts each time to train a cosine classifier, and train a total of c cosine classifiers. 62) During testing, slice the features of each query set data in the same way as during fine-tuning; 63) Input each part of the features into the corresponding classifier for prediction, and merge the prediction results of the c classifiers to obtain the small sample pest image recognition results.

2. The small sample pest image recognition method based on causal intervention technology according to claim 1 is characterized in that: The construction and training of the embedding model includes the following steps: 21) Build an embedding model; 211) ResNet-12 is used as the embedding model, which contains four residual blocks, and the number of output channels is 64, 160, 320 and 640 respectively; 212) Each residual block is set to contain three convolution operations, each of which uses a convolution kernel of size 3×3, and a normalization operation is used after each convolution operation; 213) It is set that each residual block is followed by maximum pooling, and the fourth residual block is also subjected to global average pooling, and the features output by the average pooling are used as the embedded features of the image; 214) It is set that during pre-training, according to the number of categories of the base class data, a multi-classification linear classifier is connected to predict the base class category, and a four-classification linear classifier is connected to predict the angle of image rotation; during fine-tuning, according to the number of categories of the small sample pest image, a multi-classification linear classifier is connected to fine-tune and predict the category of the small sample pest; 22) Train the embedding model; 221) Using rotation self-supervision to pre-train embedding models; In the process of training the embedding model, an additional four-category linear classifier is used to predict the rotation angle of the image, and the rotation self-supervised loss function L rotation for: L roation =L(C4(f θ (x r )),r), Among them, r is the rotation angle, x is the input image data, and x r is a picture rotated by r degrees, L is the standard cross entropy loss, f θ (x r ) is the output of the embedding model, C4 is a four-category classifier, r∈{0°, 90°, 180°, 270°}; 222) Using manifold mixture pre-trained embedding model; When training each batch of images, a network layer of the embedding model is randomly selected, and the feature maps of these data on this network layer are mixed manifolds; Characteristics of manifold mixing and tags They are: The loss function after manifold mixing is: Among them, f l (x1) and f l (x2) are the training data (x i ,y i ) and (x j ,y j ) is the feature representation of the lth layer in the embedding model; β∈[0,1]; y i and j All are one-hot labels; L is the standard cross entropy loss; g is the model representation after layer l; 223) For each batch of image data, first rotate all images by 0°, 90°, 180°, and 270°, and then perform manifold blending; For manifold blending, manifold blending is performed not only on the unrotated image, but also on the rotated image, which means that the label of the rotation angle is also transformed in the manifold blending; After each cycle of data training is completed, multiple small sample tasks are sampled using the validation set data. Similarly, each small sample task contains a small amount of support set data and a large amount of query set data. Next, the embedding model is fixed, and the support set data in the small sample task on the validation set is used to fine-tune the classifier of the last layer. Finally, the query set data in the small sample task is tested using the fine-tuned model, and the embedding model with the best test result is saved to obtain a trained embedding model. After training an embedding model, reset the random seed used during training and train a new embedding model, thereby training n embedding models.

Images