Insect pest semi-supervised target detection method combining active learning and multi-scale view learning

Abstract

The invention relates to an insect pest semi-supervised target detection method combining active learning and multi-scale view learning. The method comprises the following steps: constructing a semi-supervised target detection data set; performing data enhancement on the semi-supervised target detection data set; constructing a semi-supervised model and generating a pseudo label; performing multi-scale view learning through the student model to obtain a multi-scale view prediction result; performing semi-supervised training; performing active learning, and updating the semi-supervised target detection data set; and inputting a to-be-detected field corn image into the finally trained teacher model to obtain a final pest detection result. According to the method, active learning is combined to select some data which is beneficial to the model and can effectively improve the performance for labeling, and the labeling cost is utilized to the maximum extent; adding a multi-scale view learning structure, and capturing effective information; and finally, under the condition of limited 20% annotation data, the detection precision of more than 80% annotation data can be realized, and the annotation cost is greatly saved.

Description

Technical Field

[0001] The present invention relates to the technical field of human pest target detection, and in particular to a method for semi-supervised pest target detection combining active learning and multi-scale view learning. Background Art

[0002] Ensuring stable food production is extremely important for global food security. Crop yields are affected by many factors, among which pests are particularly significant. For example, destructive pests such as corn borers usually eat the flesh of corn leaves during the growing season, causing damage to the leaves, which will seriously affect the growth of corn crops and cause a significant drop in corn yields. In addition, the fragile corn stems and leaves are very easy to fall over. Therefore, timely detection and monitoring of pests is an important prerequisite for pest control. Early and thorough prevention and control can effectively reduce the losses caused by pests and is also the key to ensuring stable food production.

[0003] There are three main methods for detecting pest targets: manual inspection, processing methods based on machine vision, and target detection methods based on deep learning. These methods have the following limitations: The inspection effect of the first method depends entirely on human subjectivity, and the labor cost is very high and the efficiency is low. The second method is that the small target area of ​​pests is easily affected by complex backgrounds, light and other factors, the detection effect is not good, and it relies on manually designed feature extraction; the third method has been shown in studies that specific tasks in agricultural scenarios can achieve good results, but to build a model with excellent performance, a large amount of accurately labeled data is required, and the proportion of the pest area size in the picture is very small, and due to the complex environmental changes in natural fields, the complex background in the collected data brings great difficulties to manual labeling, so a semi-supervised target detection model that can be trained with a small amount of labeled data is the key to solving the problem.

[0004] The existing semi-supervised target detection annotation data selection is random, and does not consider which data annotation can bring actual and effective performance improvement. It is impossible to avoid redundant and invalid data annotation. In addition, most semi-supervised target detection methods are oriented to Microsoft's coco (Common Objects in Context) dataset, and few are designed for small target area detection of pests and diseases. Therefore, how to design a semi-supervised target detector to accurately detect small pest targets, and how to select more meaningful data for annotation under the condition of given annotation cost, is the current top priority. Summary of the invention

[0005] In order to solve the problems that the existing semi-supervised target detectors have poor detection effect on small target areas of small corn pests in complex environments, difficult feature learning and inability to label effective samples, the purpose of the present invention is to provide a method for semi-supervised target detection of pests that combines active learning and multi-scale view learning, which can effectively improve performance, maximize the use of labeling costs, capture effective information, improve detection accuracy, and greatly save labeling costs.

[0006] To achieve the above object, the present invention adopts the following technical solution: a method for semi-supervised target detection of insect pests combining active learning and multi-scale view learning, the method comprising the following steps in order:

[0007] (1) Constructing a semi-supervised object detection dataset: Collecting field corn images for preprocessing to obtain preprocessed images, randomly selecting 5% of the preprocessed images for annotation to obtain annotated images, and the annotated images and unannotated images together constitute a semi-supervised object detection dataset;

[0008] (2) Performing data enhancement on the semi-supervised target detection dataset to obtain a data-enhanced semi-supervised target detection dataset: Performing standard supervised data enhancement on the labeled data in the semi-supervised target detection dataset to obtain enhanced labeled data; Performing semi-supervised strong data enhancement and semi-supervised weak data enhancement on the unlabeled data in the semi-supervised target detection dataset to obtain strongly enhanced unlabeled data and weakly enhanced unlabeled data, respectively; the enhanced labeled data, strongly enhanced unlabeled data, and weakly enhanced unlabeled data constitute the data-enhanced semi-supervised target detection dataset;

[0009] (3) Constructing a semi-supervised model and generating pseudo labels: The semi-supervised model includes a teacher model and a student model. The weakly enhanced unlabeled data is input into the teacher model for prediction. The teacher model outputs the prediction results, and an adaptive pseudo label filter is used to generate reliable pseudo labels for unsupervised loss for the prediction results.

[0010] (4) Multi-scale view learning through the student model: downsample the strongly enhanced unlabeled data to obtain the downsampled data, combine the downsampled data and the strongly enhanced unlabeled data into two-scale view data, input the two-scale view data into the student model, and the student model outputs two view feature maps, namely the downsampled feature map and the un-downsampled feature map. After alignment and fusion, a fused feature map is obtained. The fused feature map and the downsampled feature map are input into the student model, and the student model performs forward propagation to obtain the multi-scale view prediction results;

[0011] (5) Performing semi-supervised training: inputting the enhanced labeled data into the student model to calculate the supervised loss and obtain the supervised loss; at the same time, inputting the reliable pseudo-label and multi-scale view prediction results into the teacher model to calculate the unsupervised loss and obtain the unsupervised loss; combining the supervised loss and the unsupervised loss, and back-propagating to train the student model; after training, updating the teacher model through the EMA exponential moving average algorithm to obtain an updated teacher model; the trained student model and the updated teacher model constitute a semi-supervised model after semi-supervised training;

[0012] (6) Perform active learning: input the unlabeled data into the semi-supervised model after semi-supervised training, calculate the metric value, and filter out the unlabeled data with larger values ​​according to the metric value for labeling. The labeled data is added to the labeled data part of the semi-supervised target detection dataset, and the semi-supervised target detection dataset is updated;

[0013] (7) Repeat steps (2) to (6) until the amount of labeled data in the semi-supervised target detection dataset reaches 20%, the training is completed, and the final trained teacher model is obtained. The field corn image to be detected is input into the final trained teacher model to obtain the final pest detection result.

[0014] In step (2), the standard supervised data enhancement includes random cropping and scaling, random flipping and normalization; the weak enhancement includes horizontal flipping, vertical flipping, cropping, resizing, rotation and mosaic; and the strong enhancement includes mixing and random erasing.

[0015] The step (3) specifically comprises the following steps:

[0016] (3a) The teacher model performs forward prediction on the weakly enhanced unlabeled data to obtain a prediction result, which includes a confidence score Pred tea And the corresponding target prediction box Bbox tea ; The confidence score Pred tea Sort in descending order, cumulative confidence score Pred tea Get the filter bottom limit index According to the filter bottom index Get the threshold τ l , filter out the confidence scores less than τ l The confidence score and the corresponding target prediction box are used to obtain the filtered confidence score Pred′ tea And the corresponding target prediction box Bbox′ tea :

[0017]

[0018] Pred′ tea =Predtea [Pred tea >τ l ]

[0019] Among them, sort means to sort Pred tea Sort in descending order, is the confidence score after descending sorting; sum represents the Accumulation;

[0020] The confidence score Pred′ after screening tea Store into a teacher prediction cache Scores of size 100;

[0021] (3b) The confidence score Pred′ after screening tea Operate separately by category: Pred′ for each category tea Contains bimodal data of positive and negative modes, and performs Gaussian mixture modeling on cached Scores[c]:

[0022]

[0023] in, represents the Gaussian distribution of the negative mode, represents the Gaussian distribution of the positive mode, are the weight, mean and variance of the negative mode, are the weight, mean and variance of the positive mode, c is the total number of categories contained in the semi-supervised target detection dataset; s c Represents the value in the confidence score cache of the category, P(s c ) represents the probability distribution of the confidence buffer containing positive and negative modes;

[0024] Infer the posterior using the EM expectation-maximization algorithm Represents the probability of setting the corresponding target prediction box as a pseudo label according to the confidence score, and determines the adaptive score threshold for each category

[0025]

[0026] use Pred tea Filter to get category pseudo labels The positioning pseudo label is obtained by filtering the soft teacher's jitter screening method Category pseudo-labeling and localization pseudo-labels Composing reliable pseudo labels

[0027] In the step (4), downsampling the strongly enhanced unlabeled data to obtain downsampled data, the downsampled data and the strongly enhanced unlabeled data are combined into two-scale view data, the two-scale view data are input into the student model, and the student model outputs two view feature maps, namely the downsampled feature map and the un-downsampled feature map. After alignment and fusion, obtaining the fused feature map specifically includes the following steps:

[0028] (4a) For the strongly enhanced unlabeled data img unlabeled Perform 0.5 times downsampling to obtain downsampled data

[0029]

[0030] In the formula, downsample is downsampling;

[0031] The downsampled data And the unlabeled data img after strong enhancement unlabeled The two scales of view data are input into the feature extraction structure of the student model. After that, we get two view feature maps, namely the unsampled feature map X and the downsampled feature map X d :

[0032]

[0033] Downsample feature map X d Both the un-downsampled feature map X contain 5 layers of feature maps from deep to shallow;

[0034] (4b) Align the two view feature maps to obtain the aligned feature map X′ d :

[0035] X′ d =align(X d ,X)

[0036] In the formula, align means alignment;

[0037] The aligned feature map X′ d Fuse with the unsampled feature map X to obtain the fused feature map X * :

[0038] X * [i] = ECA Fusion (X ′ d [i],X[i])

[0039] In the formula, ECA Fusion represents fusion.

[0040] In step (5), the enhanced labeled data is input into the student model to calculate the supervised loss to obtain the supervised loss; at the same time, the reliable pseudo-label and the multi-scale view prediction results are input into the teacher model to calculate the unsupervised loss to obtain the unsupervised loss; the combined supervised loss and unsupervised loss specifically include the following steps:

[0041] (5a) Input the enhanced labeled data into the student model to calculate the supervision loss and obtain the supervision loss Loss sup :

[0042]

[0043] Among them, Loss cls is the cross entropy loss, Loss reg is the L1 loss, T() is the standard supervised data augmentation, N is the number of samples in the supervised data, and f stu () is the student model; is the labeled data, y n is the label of the annotated data;

[0044] (5b) Input the reliable pseudo-label and multi-scale view prediction results into the teacher model for unsupervised loss calculation to obtain the unsupervised loss Loss un sup :

[0045]

[0046] Among them, T′() is semi-supervised strong data enhancement, Loss cls is the loss of the soft teacher, M is the number of samples of unlabeled data, is a reliable pseudo label, is the mth unlabeled data;

[0047] (5c) Calculate the total loss Loss:

[0048] Loss=Loss sup +λ unsup Loss un sup

[0049] In the formula, λ unsup is the unsupervised loss weight, λ unsup The value of is 2.0, and the loss is back-propagated to the student model to obtain the updated student model after the tth round, and the teacher model is updated by the EMA exponential moving average algorithm to obtain the updated teacher model:

[0050]

[0051] in, is the updated parameter of the teacher model after the tth round, are the parameters of the teacher model in round t-1, It is the updated parameter of the student model in round t.

[0052] The step (6) specifically comprises the following steps:

[0053] (6a) Use the semi-supervised model after semi-supervised training to perform metric calculations on unlabeled data;

[0054] (6b) Calculate the information metric of each graph, including Divergence, Meta Distance, and Uncertainty, and then perform L2 regularization on these three metrics to obtain the information value of each graph;

[0055] (6c) Then the information value of each image is sorted in descending order, and the first α unlabeled data are selected and added to the labeled data set, where α refers to 2.5% of all collected data, thus forming a new semi-supervised object detection data set for a new round of training.

[0056] The step (4b) specifically comprises the following steps:

[0057] (4b1) For the downsampled feature map X d , discard its fifth feature map X d [5] Then the first feature map X d [1] After upsampling by 2 times, we get X′ d [1], then X′ d [1] Added to the head position of the feature group to form a downsampled view feature group X′ aligned with the normal view d [i]:

[0058] X′ d [1] = upsample(X d [1])

[0059] X′ d [i] = X d [i-1],1 <i≤5

[0060] (4b2) After alignment, X′ d And X perform ECA efficient channel attention fusion, the specific formula is as follows:

[0061] X1[i]=GAP(X[i])

[0062] X2[i]=GAP(X′ d [i])

[0063] X 12 [i]=Cat(X1[i],X2[i])

[0064] X 12 [i] = Conv1×1(X 12 [i])

[0065] Probs 12 [i] = Softmax(X 12 [i])

[0066] X * [i] = Probs 12 [i][1]*X[i]+Probs 12 [i][2]*X′ d [i]

[0067] Among them, GAP is global average pooling; Cat is tensor concatenation; Conv1×1 is a one-dimensional convolution with a convolution kernel size of 3; X1[i] is the result of global average pooling of X[i], X[i] is the feature map of unlabeled data without downsampling, and X2[i] is the result of X ′ d [i] The result after GAP, X 12 [i] is the result of concatenating X1[i] and X2[i], Probs 12 [i] is the fusion weight, X * [i] is X′ d The fused feature map result obtained after ECA efficient channel attention fusion with X.

[0068] The step (6b) specifically comprises the following steps:

[0069] (6b1) Calculate the teacher-student divergence: First, the unlabeled data in the semi-supervised object detection dataset is input into the teacher model without any enhancement to obtain the detection output bboxs tea ,labels tea :

[0070] bboxs tea ,labels tea =f tea (img unlabeled )

[0071] In the formula, bboxs tea is the teacher model f tea () for unlabeled data img unlabeled Target prediction box obtained by forward prediction, labels tea is bboxs tea The corresponding predicted category label;

[0072] Then input the unlabeled data into the student model and get the corresponding student model prediction:

[0073] bboxs stu ,labels stu =f stu (img unlabeled )

[0074] In the formula, bboxs stu is the student model f stu () for unlabeled data img unlabeled Target prediction box obtained by forward prediction, labels stu is bboxs stu The corresponding predicted category label;

[0075] The teacher model is then regarded as the true label to calculate the average precision mAP of the student and the teacher. The teacher-student divergence is calculated based on the average precision mAP:

[0076] Divergence = 1-mAP(bboxs tea ,labels tea ,bboxs stu ,labels stu )

[0077] (6b2) Calculate Meta Distance: Use the labeled data to calculate the region of interest feature ROIFeat for each class during the supervised loss calculation, and update the meta feature Meta Feat for each class c :

[0078]

[0079] ROI Features c =mean(ROI Feats,c)

[0080]

[0081] Among them, GAP is global average pooling, GT is the target prediction box of the labeled data, roi_extractor is a method for extracting features from the feature map of the region of interest ROI, and mean() is the average; Represents the backbone and feature enhancement structure in the student model; img labeled It is the labeled data, ROI Feats c represents the ROI feature of the cth class, is the c-th meta-feature after the t-th round of training update;

[0082] Calculate Meta Distance:

[0083]

[0084]

[0085] Among them, consine() is cosine similarity; ROI Feat unlabeled Meta Feat is the ROI feature of all categories extracted from unlabeled data; c Meta distance is the meta feature of the current c-th class; c is the meta-deviation of the cth class; It is the ROI feature of the cth category extracted from unlabeled data; Represents the backbone and feature enhancement structure in the teacher model;

[0086] The Meta Distance of the final image = mean (Meta distance c );

[0087] (6b3) Calculate the uncertainty:

[0088] Get the unprocessed predictions cls_pred, bbox_pred:

[0089]

[0090] in, It is the structure of the teacher model from its backbone network to the ROI detection head. cls_pred is the class distribution probability of each prediction result, and bbox_pred is the bounding box of each prediction result.

[0091] Calculate the class distribution probability entropy of each prediction result, and average the entropy of each prediction result to get img unlabeled Uncertainty:

[0092]

[0093] Where B is img unlabeled The number of target prediction boxes predicted in cls_pred(c i ; b j ) is the jth target prediction box b j Belongs to the i-th category c i The probability of

[0094] (6b4) Perform L2 regularization on the three metrics calculated above:

[0095]

[0096] Where Info is the information value of the entire graph.

[0097] It can be seen from the above technical scheme that the beneficial effects of the present invention are: first, in view of the defect that the selection of annotation data for general semi-supervised target detection is random, the present invention combines active learning to select some data that are beneficial to the model and can effectively improve the performance for annotation, so as to maximize the use of annotation costs; second, the general semi-supervised detection framework is improved, a multi-scale view learning structure is added, and the consistency learning of shallow features and deep semantics of pest and disease targets is strengthened to capture effective information; third, in the pseudo-label screening part, a Gaussian mixture model adaptive filter is introduced to avoid the shortcomings of insufficient pseudo-labels of pest and disease targets in the early stage and the introduction of deviations in the later stage, and finally, with a limited 20% of labeled data, the detection accuracy of more than 80% of the labeled data can be achieved, which greatly saves the labeling cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] Figure 1 is a flow chart of the method of the present invention;

[0099] Figure 2 A schematic diagram of a multi-scale view learning structure of the present invention;

[0100] Figure 3 It is a schematic diagram of the active learning structure of the present invention;

[0101] Figure 4 It is a schematic diagram of the detection results obtained by the present invention. DETAILED DESCRIPTION

[0102] like Figure 1 As shown, a method for semi-supervised target detection of insect pests combining active learning and multi-scale view learning is provided, and the method comprises the following steps in sequence:

[0103] (1) Constructing a semi-supervised object detection dataset: Collecting field corn images for preprocessing to obtain preprocessed images, randomly selecting 5% of the preprocessed images for annotation to obtain annotated images, and the annotated images and unannotated images together constitute a semi-supervised object detection dataset;

[0104] (2) Performing data enhancement on the semi-supervised target detection dataset to obtain a data-enhanced semi-supervised target detection dataset to achieve consistency regularization and pseudo-label learning during the training process: Performing standard supervised data enhancement on the labeled data in the semi-supervised target detection dataset to obtain enhanced labeled data; Performing semi-supervised strong data enhancement and semi-supervised weak data enhancement on the unlabeled data in the semi-supervised target detection dataset to obtain strongly enhanced unlabeled data and weakly enhanced unlabeled data respectively; The enhanced labeled data, the strongly enhanced unlabeled data, and the weakly enhanced unlabeled data constitute the data-enhanced semi-supervised target detection dataset;

[0105] (3) constructing a semi-supervised model and generating pseudo labels, so as to dynamically screen the current prediction results and obtain reliable pseudo labels: the semi-supervised model includes a teacher model and a student model, the weakly enhanced unlabeled data is input into the teacher model for prediction, the teacher model outputs the prediction results, and an adaptive pseudo label filter is used to generate reliable pseudo labels for unsupervised loss for the prediction results;

[0106] (4) Multi-scale view learning through the student model: downsample the strongly enhanced unlabeled data to obtain the downsampled data, combine the downsampled data and the strongly enhanced unlabeled data into two-scale view data, input the two-scale view data into the student model, and the student model outputs two view feature maps, namely the downsampled feature map and the un-downsampled feature map. After alignment and fusion, a fused feature map is obtained. The fused feature map and the downsampled feature map are input into the student model, and the student model performs forward propagation to obtain the multi-scale view prediction results;

[0107] (5) Performing semi-supervised training: inputting the enhanced labeled data into the student model to calculate the supervised loss and obtain the supervised loss; at the same time, inputting the reliable pseudo-label and multi-scale view prediction results into the teacher model to calculate the unsupervised loss and obtain the unsupervised loss; combining the supervised loss and the unsupervised loss, and back-propagating to train the student model; after training, updating the teacher model through the EMA exponential moving average algorithm to obtain an updated teacher model; the trained student model and the updated teacher model constitute a semi-supervised model after semi-supervised training;

[0108] (6) Perform active learning: input the unlabeled data into the semi-supervised model after semi-supervised training, calculate the metric value, and filter out the unlabeled data with larger values ​​according to the metric value for labeling. The labeled data is added to the labeled data part of the semi-supervised target detection dataset, and the semi-supervised target detection dataset is updated;

[0109] (7) Repeat steps (2) to (6) until the amount of labeled data in the semi-supervised target detection dataset reaches 20%, so that more accurate corn pest detection results can be obtained under limited labeling costs. After training is completed, the final trained teacher model is obtained, and the field corn image to be detected is input into the final trained teacher model to obtain the final pest detection result, such as Figure 4 shown.

[0110] In step (2), the standard supervised data enhancement includes random cropping and scaling, random flipping and normalization; the weak enhancement includes horizontal flipping, vertical flipping, cropping, resizing, rotation and mosaic; and the strong enhancement includes mixing and random erasing.

[0111] The step (3) specifically comprises the following steps:

[0112] (3a) The teacher model performs forward prediction on the weakly enhanced unlabeled data to obtain a prediction result, which includes a confidence score Pred tea And the corresponding target prediction box Bbox tea ; The confidence score Pred tea Sort in descending order, cumulative confidence score Pred tea Get the filter bottom limit index According to the filter bottom index Get the threshold τ l , filter out the confidence scores less than τ l The confidence score and the corresponding target prediction box are used to obtain the filtered confidence score Pred′ tea And the corresponding target prediction box Bbox′ tea :

[0113]

[0114] Pred′ tea =Pred tea [Pred tea >τ l ]

[0115] Among them, sort means to sort Pred tea Sort in descending order, is the confidence score after descending sorting; sum represents the Accumulation; The purpose of the 0.5 is to prevent the cumulative sum from being too small, which would result in all predictions being filtered out.

[0116] The confidence score Pred′ after screening tea Store into a teacher prediction cache Scores of size 100;

[0117] (3b) The confidence score Pred′ after screening tea Operate separately by category: Pred′ for each category tea Contains bimodal data of positive and negative modes, and performs Gaussian mixture modeling on cached Scores[c]:

[0118]

[0119] in, represents the Gaussian distribution of the negative mode, represents the Gaussian distribution of the positive mode, are the weight, mean and variance of the negative mode, are the weight, mean and variance of the positive mode, c is the total number of categories contained in the semi-supervised target detection dataset; s c Represents the value in the confidence score cache of the category, P(s c ) represents the probability distribution of the confidence buffer containing positive and negative modes;

[0120] Infer the posterior using the EM expectation-maximization algorithm Represents the probability of setting the corresponding target prediction box as a pseudo label according to the confidence score, and determines the adaptive score threshold for each category

[0121]

[0122] use Pred tea Filter to get category pseudo labels The positioning pseudo label is obtained by filtering the soft teacher's jitter screening method Category pseudo-labeling and localization pseudo-labels Composing reliable pseudo labels

[0123] like Figure 2 As shown, in the step (4), the unlabeled data after strong enhancement is downsampled to obtain the downsampled data, the downsampled data and the unlabeled data after strong enhancement are combined into two-scale view data, the two-scale view data are input into the student model, and the student model outputs two view feature maps, namely the downsampled feature map and the un-downsampled feature map. After alignment and fusion, the fused feature map is obtained. Specifically, the following steps are included:

[0124] (4a) For the strongly enhanced unlabeled data img unlabeled Perform 0.5 times downsampling to obtain downsampled data

[0125]

[0126] In the formula, downsample is downsampling;

[0127] The downsampled data And the unlabeled data img after strong enhancement unlabeled The two scales of view data are input into the feature extraction structure of the student model. After that, we get two view feature maps, namely the unsampled feature map X and the downsampled feature map X d :

[0128]

[0129] Downsample feature map X d Both the un-downsampled feature map X contain 5 layers of feature maps from deep to shallow;

[0130] (4b) Align the two view feature maps to obtain the aligned feature map X′ d :

[0131] X′ d =align(X d ,X)

[0132] In the formula, align means alignment;

[0133] The aligned feature map X′ d Fuse with the unsampled feature map X to obtain the fused feature map X * :

[0134] X * [i] = ECA Fusion (X ′ d [i],X[i])

[0135] In the formula, ECA Fusion represents fusion.

[0136] In step (5), the enhanced labeled data is input into the student model to calculate the supervised loss to obtain the supervised loss; at the same time, the reliable pseudo-label and the multi-scale view prediction results are input into the teacher model to calculate the unsupervised loss to obtain the unsupervised loss; the combined supervised loss and unsupervised loss specifically include the following steps:

[0137] (5a) Input the enhanced labeled data into the student model to calculate the supervision loss and obtain the supervision loss Loss sup :

[0138]

[0139] Among them, Losscls is the cross entropy loss, Loss reg is the L1 loss, T() is the standard supervised data augmentation, N is the number of samples in the supervised data, and f stu () is the student model; is the labeled data, y n is the label of the annotated data;

[0140] (5b) Input the reliable pseudo-label and multi-scale view prediction results into the teacher model for unsupervised loss calculation to obtain the unsupervised loss Loss un sup :

[0141]

[0142] Among them, T′() is semi-supervised strong data enhancement, Loss cls is the loss of the soft teacher, M is the number of samples of unlabeled data, is a reliable pseudo label, is the mth unlabeled data;

[0143] (5c) Calculate the total loss Loss:

[0144] Loss=Loss sup +λ unsup Loss unsup

[0145] In the formula, λ unsup is the unsupervised loss weight, λ unsup The value of is 2.0, and the loss is back-propagated to the student model to obtain the updated student model after the tth round, and the teacher model is updated by the EMA exponential moving average algorithm to obtain the updated teacher model:

[0146]

[0147] in, is the updated parameter of the teacher model after the tth round, are the parameters of the teacher model in round t-1, It is the updated parameter of the student model in round t.

[0148] like Figure 3 As shown, the step (6) specifically includes the following steps:

[0149] (6a) Use the semi-supervised model after semi-supervised training to perform metric calculations on unlabeled data;

[0150] (6b) Calculate the information metric of each graph, including Divergence, Meta Distance, and Uncertainty, and then perform L2 regularization on these three metrics to obtain the information value of each graph;

[0151] (6c) Then the information value of each image is sorted in descending order, and the first α unlabeled data are selected and added to the labeled data set, where α refers to 2.5% of all collected data, thus forming a new semi-supervised object detection data set for a new round of training.

[0152] The step (4b) specifically comprises the following steps:

[0153] (4b1) For the downsampled feature map X d , discard its fifth feature map X d [5] Then the first feature map X d [1] After upsampling by 2 times, we get X′ d [1], then X′ d [1] Added to the head position of the feature group to form a downsampled view feature group X′ aligned with the normal view d [i]:

[0154] X′ d [1] = upsample(X d [1])

[0155] X′ d [i] = X d [i-1],1 <i≤5

[0156] (4b2) After alignment, X′ d And X perform ECA efficient channel attention fusion, the specific formula is as follows:

[0157] X1[i]=GAP(X[i])

[0158] X2[i]=GAP(X′ d [i])

[0159] X 12 [i]=Cat(X1[i],X2[i])

[0160] X 12 [i] = Conv1×1(X 12 [i])

[0161] Probs 12 [i] = Softmax(X 12 [i])

[0162] X *[i] = Probs 12 [i][1]*X[i]+Probs 12 [i][2]*X′ d [i]

[0163] Among them, GAP is global average pooling; Cat is tensor concatenation; Conv1×1 is a one-dimensional convolution with a convolution kernel size of 3; X1[i] is the result of global average pooling of X[i], X[i] is the feature map of unlabeled data without downsampling, and X2[i] is X′ d [i] The result after GAP, X 12 [i] is the result of concatenating X1[i] and X2[i], Probs 12 [i] is the fusion weight, X * [i] is X′ d The fused feature map result obtained after ECA efficient channel attention fusion with X.

[0164] The step (6b) specifically comprises the following steps:

[0165] (6b1) Calculate the teacher-student divergence: First, the unlabeled data in the semi-supervised object detection dataset is input into the teacher model without any enhancement to obtain the detection output bboxs tea ,labels tea :

[0166] bboxs tea ,labels tea =f tea (img unlabeled )

[0167] In the formula, bboxs tea is the teacher model f tea () for unlabeled data img unlabeled Target prediction box obtained by forward prediction, labels tea is bboxs tea The corresponding predicted category label;

[0168] Then input the unlabeled data into the student model and get the corresponding student model prediction:

[0169] bboxs stu ,labels stu =f stu (img unlabeled )

[0170] In the formula, bboxs stu is the student model f stu() for unlabeled data img unlabeled Target prediction box obtained by forward prediction, labels stu is bboxs stu The corresponding predicted category label;

[0171] The teacher model is then regarded as the true label to calculate the average precision mAP of the student and the teacher. The teacher-student divergence is calculated based on the average precision mAP:

[0172] Divergence = 1-mAP(bboxs tea ,labels tea ,bboxs stu ,labels stu )

[0173] (6b2) Calculate Meta Distance: Use the labeled data to calculate the region of interest feature ROIFeat for each class during the supervised loss calculation, and update the meta feature Meta Feat for each class c :

[0174]

[0175] ROI Features c =mean(ROI Feats,c)

[0176]

[0177] Among them, GAP is global average pooling, CT is the target prediction box of the labeled data, roi_extractor is a method for extracting features from the feature map of the region of interest ROI, and mean() is the average; Represents the backbone and feature enhancement structure in the student model; img labeled It is the labeled data, ROI Feats c represents the ROI feature of the cth class, is the c-th meta-feature after the t-th round of training update;

[0178] Calculate Meta Distance:

[0179]

[0180]

[0181] Among them, consine() is cosine similarity; ROI Feat unlabeled Meta Feat is the ROI feature of all categories extracted from unlabeled data;c Meta distance is the meta feature of the current c-th class; c is the meta-deviation of the cth class; It is the ROI feature of the cth category extracted from unlabeled data; Represents the backbone and feature enhancement structure in the teacher model;

[0182] The Meta Distance of the final image = mean (Meta distance c );

[0183] (6b3) Calculate the uncertainty:

[0184] Get the unprocessed predictions cls_pred, bbox_pred:

[0185]

[0186] in, It is the structure of the teacher model from its backbone network to the ROI detection head. cls_pred is the class distribution probability of each prediction result, and bbox_pred is the bounding box of each prediction result.

[0187] Calculate the class distribution probability entropy of each prediction result, and average the entropy of each prediction result to get img unlabeled Uncertainty:

[0188]

[0189] Where B is img unlabeled The number of target prediction boxes predicted in cls_pred(c i ; b j ) is the jth target prediction box b j Belongs to the i-th category c i The probability of

[0190] (6b4) Perform L2 regularization on the three metrics calculated above:

[0191]

[0192] Where Info is the information value of the entire graph.

[0193] like Figure 4 As shown in the figure, the blue bounding box represents the pest area target detected by the model, corn refers to the category of the target, that is, the corn pest area, and the number is the confidence score of the model for the detected target. The larger the score, the more certain the model is that this is a pest area target.

[0194] In summary, the present invention combines active learning to select some data that are beneficial to the model and can effectively improve the performance for annotation, so as to maximize the use of annotation costs; improve the general semi-supervised detection framework, add a multi-scale view learning structure, strengthen the consistency learning of shallow features and deep semantics of pest targets, and capture effective information; in the pseudo-label screening part, introduce a Gaussian mixture model adaptive filter, which can avoid the shortcomings of insufficient pseudo-labels of pest targets in the early stage and the introduction of deviations in the later stage, and finally achieve a detection accuracy of more than 80% of the labeled data under the limited 20% labeled data, which greatly saves the labeling cost.

Claims

1. A semi-supervised target detection method for insect pests combining active learning and multi-scale view learning, characterized by: The method comprises the following steps in order: (1) Constructing a semi-supervised object detection dataset: Collect field corn images for preprocessing to obtain preprocessed images, randomly select 5% of the preprocessed images for annotation, and obtain annotated images. The annotated images and unannotated images together constitute a semi-supervised object detection dataset; (2) Performing data enhancement on the semi-supervised target detection dataset to obtain a data-enhanced semi-supervised target detection dataset: Performing standard supervised data enhancement on the labeled data in the semi-supervised target detection dataset to obtain enhanced labeled data; The unlabeled data in the semi-supervised target detection dataset are subjected to semi-supervised strong data enhancement and semi-supervised weak data enhancement, respectively, to obtain strongly enhanced unlabeled data and weakly enhanced unlabeled data; The enhanced labeled data, the strongly enhanced unlabeled data, and the weakly enhanced unlabeled data constitute the data-enhanced semi-supervised object detection dataset; (3) Constructing a semi-supervised model and generating pseudo labels: The semi-supervised model includes a teacher model and a student model. The weakly enhanced unlabeled data is input into the teacher model for prediction. The teacher model outputs the prediction results, and an adaptive pseudo label filter is used to generate reliable pseudo labels for unsupervised loss for the prediction results. (4) Multi-scale view learning through the student model: downsample the strongly enhanced unlabeled data to obtain the downsampled data, combine the downsampled data and the strongly enhanced unlabeled data into two-scale view data, input the two-scale view data into the student model, and the student model outputs two view feature maps, namely the downsampled feature map and the un-downsampled feature map. After alignment and fusion, a fused feature map is obtained. The fused feature map and the downsampled feature map are input into the student model, and the student model performs forward propagation to obtain the multi-scale view prediction results; (5) Performing semi-supervised training: inputting the enhanced labeled data into the student model to calculate the supervised loss and obtain the supervised loss; at the same time, inputting the reliable pseudo-label and multi-scale view prediction results into the teacher model to calculate the unsupervised loss and obtain the unsupervised loss; combining the supervised loss and the unsupervised loss, and back-propagating to train the student model; after training, updating the teacher model through the EMA exponential moving average algorithm to obtain an updated teacher model; the trained student model and the updated teacher model constitute a semi-supervised model after semi-supervised training; (6) Perform active learning: input the unlabeled data into the semi-supervised model after semi-supervised training, calculate the metric value, and filter out the unlabeled data with larger values ​​according to the metric value for labeling. The labeled data is added to the labeled data part of the semi-supervised target detection dataset, and the semi-supervised target detection dataset is updated; (7) Repeat steps (2) to (6) until the amount of labeled data in the semi-supervised target detection dataset reaches 20%, the training is completed, and the final trained teacher model is obtained. The field corn image to be detected is input into the final trained teacher model to obtain the final pest detection result.

2. The method for semi-supervised target detection of insect pests combining active learning and multi-scale view learning according to claim 1, characterized in that: In step (2), the standard supervised data enhancement includes random cropping and scaling, random flipping and normalization; the weak enhancement includes horizontal flipping, vertical flipping, cropping, resizing, rotation and mosaic; The strong enhancement includes mixing and random erasing.

3. The method for semi-supervised target detection of insect pests combining active learning and multi-scale view learning according to claim 1, characterized in that: The step (3) specifically comprises the following steps: (3a) The teacher model performs forward prediction on the weakly enhanced unlabeled data to obtain a prediction result, which includes a confidence score Pred tea And the corresponding target prediction box Bbox tea ; The confidence score Pred tea Sort in descending order, cumulative confidence score Pred tea Get the filter bottom limit index According to the filter bottom index Get the threshold τ l , filter out the confidence scores less than τ l The confidence score and the corresponding target prediction box are used to obtain the filtered confidence score Pred′ tea And the corresponding target prediction box Bbox′ tea : Previous tea =Prev tea [Previous tea >τ l ] Among them, sort means to sort Pred tea Sort in descending order, is the confidence score after descending sorting; sum represents the Accumulation; The confidence score Pred′ after screening tea Store into a teacher prediction cache Scores of size 100; (3b) The confidence score Pred′ after screening dea Operate separately by category: Pred′ for each category dea Contains bimodal data of positive and negative modes, and performs Gaussian mixture modeling on cached Scores[c]: in, represents the Gaussian distribution of the negative mode, represents the Gaussian distribution of the positive mode, are the weight, mean and variance of the negative mode, are the weight, mean and variance of the positive mode, c is the total number of categories contained in the semi-supervised target detection dataset; s c Represents the value in the confidence score cache of the category, P(s c ) represents the probability distribution of the confidence buffer containing positive and negative modes; Infer the posterior using the EM expectation-maximization algorithm Represents the probability of setting the corresponding target prediction box as a pseudo label according to the confidence score, and determines the adaptive score threshold for each category use Pred tea Filter to get category pseudo labels The positioning pseudo label is obtained by filtering the soft teacher's jitter screening method Category pseudo-labels and localization pseudo-labels Composing reliable pseudo labels 4. The method for semi-supervised target detection of insect pests combining active learning and multi-scale view learning according to claim 1, characterized in that: In the step (4), downsampling the strongly enhanced unlabeled data to obtain downsampled data, the downsampled data and the strongly enhanced unlabeled data are combined into two-scale view data, the two-scale view data are input into the student model, and the student model outputs two view feature maps, namely the downsampled feature map and the un-downsampled feature map. After alignment and fusion, obtaining the fused feature map specifically includes the following steps: (4a) For the strongly enhanced unlabeled data img unlabeled Perform 0.5 times downsampling to obtain downsampled data In the formula, downsample is downsampling; The downsampled data And the unlabeled data img after strong enhancement unlabeled The two scales of view data are input into the feature extraction structure of the student model. After that, we get two view feature maps, namely the unsampled feature map X and the downsampled feature map X d : Downsample feature map X d Both the un-downsampled feature map X contain 5 layers of feature maps from deep to shallow; (4b) Align the two view feature maps to obtain the aligned feature map X′ d : X′ d =align(X d ,X) In the formula, align means alignment; The aligned feature map X′ d Fuse with the unsampled feature map X to obtain the fused feature map X * : X * [i]=ECAFusion(X′ d [i],X[i]) Where, ECAFusion represents fusion.

5. The method for semi-supervised target detection of insect pests combining active learning and multi-scale view learning according to claim 1, characterized in that: In step (5), the enhanced labeled data is input into the student model to calculate the supervised loss to obtain the supervised loss; at the same time, the reliable pseudo-label and the multi-scale view prediction results are input into the teacher model to calculate the unsupervised loss to obtain the unsupervised loss; the combined supervised loss and unsupervised loss specifically include the following steps: (5a) Input the enhanced labeled data into the student model to calculate the supervision loss and obtain the supervision loss Loss sup : Among them, Loss cls is the cross entropy loss, Loss reg is the L1 loss, T() is the standard supervised data augmentation, N is the number of samples in the supervised data, and f stu () is the student model; is the labeled data, y n is the label of the annotated data; (5b) Input the reliable pseudo-label and multi-scale view prediction results into the teacher model for unsupervised loss calculation to obtain the unsupervised loss Loss unsup : Among them, T′() is semi-supervised strong data enhancement, Loss cls is the loss of the soft teacher, M is the number of samples of unlabeled data, is a reliable pseudo label, is the mth unlabeled data; (5c) Calculate the total loss Loss: Loss=Loss sup +λ unsup Loss unsup In the formula, λ unsup is the unsupervised loss weight, λ unsup The value of is 2.0, and the loss is back-propagated to the student model to obtain the updated student model after the tth round, and the teacher model is updated by the EMA exponential moving average algorithm to obtain the updated teacher model: in, is the updated parameter of the teacher model after the tth round, are the parameters of the teacher model in round t-1, It is the updated parameter of the student model in round t.

6. The method for semi-supervised target detection of insect pests combining active learning and multi-scale view learning according to claim 1, characterized in that: The step (6) specifically comprises the following steps: (6a) Use the semi-supervised model after semi-supervised training to perform metric calculations on unlabeled data; (6b) Calculate the information metric of each image, including teacher-student disagreement, meta distance, and uncertainty, and then perform L2 regularization on these three metrics to obtain the information value of each image; (6c) Then the information value of each image is sorted in descending order, and the first α unlabeled data are selected and added to the labeled data set, where α refers to 2.5% of all collected data, thus forming a new semi-supervised object detection data set for a new round of training.

7. The method for semi-supervised target detection of insect pests combining active learning and multi-scale view learning according to claim 4, characterized in that: The step (4b) specifically comprises the following steps: (4b1) For the downsampled feature map X d , discard its fifth feature map X d [5] Then the first feature map X d [1] After upsampling by 2 times, we get X′ d [1], then X′ d [1] Added to the head position of the feature group to form a downsampled view feature group X′ aligned with the normal view d [i]: X′ d [1]=upsample(X d [1]) X′ d [i]=X d [i-1],1<i≤5 (4b2) After alignment, X′ d And X perform ECA efficient channel attention fusion, the specific formula is as follows: X1[i]=GAP(X[i]) X2[i]=GAP(X′ d [i]) X 12 [i]=Cat(X1[i],X2[i]) X 12 [i]=Conv1×1(X 12 [i]) Probs 12 [i]=Softmax(X 12 [i]) X * [i]=Probs 12 [i][1]*X[i]+Probs 12 [i][2]*X′ d [i] Among them, GAP is global average pooling; Cat is tensor concatenation; Conv1×1 is a one-dimensional convolution with a convolution kernel size of 3; X1[i] is the result of global average pooling of X[i], X[i] is the feature map of unlabeled data without downsampling, and X2[i] is X′ d [i] The result after GAP, X 12 [i] is the result of concatenating X1[i] and X2[i], Probs 12 [i] is the fusion weight, X * [i] is X′ d The fused feature map result obtained after ECA efficient channel attention fusion with X.

8. The method for semi-supervised target detection of insect pests combining active learning and multi-scale view learning according to claim 6, characterized in that: The step (6b) specifically comprises the following steps: (6b1) Calculate the teacher-student divergence: First, the unlabeled data in the semi-supervised object detection dataset is input into the teacher model without any enhancement to obtain the detection output bboxs tea , labels tea : bboxs tea ,labels tea =f tea (img unlabeled ) In the formula, bboxs tea is the teacher model f tea () for unlabeled data img unlabeled Target prediction box obtained by forward prediction, labels tea is bboxs tea The corresponding predicted category label; Then input the unlabeled data into the student model and get the corresponding student model prediction: bboxs stu ,labels stu =f stu (img unlabeled ) In the formula, bboxs stu is the student model f stu () for unlabeled data img unlabeled Target prediction box obtained by forward prediction, labels stu is bboxs stu The corresponding predicted category label; The teacher model is then regarded as the true label to calculate the average precision mAP of the student and the teacher. The teacher-student divergence is calculated based on the average precision mAP: Divergence=1-mAP(bboxs tea ,labels tea ,bboxs stu ,labels stu ) (6b2) Calculate Meta Distance: Use the labeled data to calculate the region of interest feature ROIFeat for each class during the supervised loss calculation, and update the meta feature Meta Feat for each class c : ROI Feats c =mean(ROI Feats,c) Among them, GAP is global average pooling, GT is the target prediction box of the labeled data, roi_extractor is a method for extracting features from the feature map of the region of interest ROI, and mean() is the average; Represents the backbone and feature enhancement structure in the student model; img labeled It is the labeled data, ROI Feats c represents the ROI feature of the cth class, is the c-th meta-feature after the t-th round of training update; Calculate Meta Distance: Among them, consine() is cosine similarity; ROI Feat unlabeled Meta Feat is the ROI feature of all categories extracted from unlabeled data; c Meta distance is the meta feature of the current c-th class; c is the meta-deviation of class c; ROIFeat c unlabeled It is the ROI feature of the cth category extracted from unlabeled data; Represents the backbone and feature enhancement structure in the teacher model; The Meta Distance of the final image = mean (Meta distance c ); (6b3) Calculate the uncertainty: Get the unprocessed predictions cls_pred, bbox_pred: in, It is the structure of the teacher model from its backbone network to the ROI detection head. cls_pred is the class distribution probability of each prediction result, and bbox_pred is the bounding box of each prediction result. Calculate the class distribution probability entropy of each prediction result, and average the entropy of each prediction result to get img unlabeled Uncertainty: Where B is img unlabeled The number of target prediction boxes predicted in cls_pred(c i ; b j ) is the jth target prediction box b j Belongs to the i-th category c i probability; (6b4) Perform L2 regularization on the three metrics calculated above: Where Info is the information value of the entire graph.

Images