A small sample-based target detection method

Abstract

The application discloses a kind of target detection methods based on small sample, comprising the following steps: S1: image acquisition;S2: image pre-processing;S3: construct small sample target detection model;S4: respectively using base class data set and new class data set to small sample target detection model is trained in two stages;S5: the network model obtained using step S4 is tested.This method reduces the dependence of target detection algorithm on a large number of training data, alleviates the problem of difficult data acquisition and high cost of manually labeled data, alleviates the contradiction between different modules that affect each other and reduce performance, improves the detection performance without increasing the training cost;Reduce the detection result error caused by classification error, improve the classification performance of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and deep learning, and particularly relates to a target detection method based on small samples. BACKGROUND

[0002] Target detection based on digital images is a key technical link for many automatic applications. Traditional detection methods mainly rely on image processing, which often needs more human intervention to select appropriate features to achieve accurate extraction of targets, and the work is complex and computationally intensive. In recent years, deep learning algorithms represented by two-stage R-CNN series and one-stage YOLO series have made major breakthroughs in target detection tasks, and have greatly improved the accuracy of target detection and classification tasks, and have good performance. However, these methods often rely on a large amount of labeled data, and the cost of data acquisition is high. Especially for some applications, such as underwater organisms and orchard fruits, collecting training data and labeled data requires a lot of time and labor cost.

[0003] Based on the problems existing in this kind of detection scene, small sample target detection is getting more and more attention. The purpose of small sample target detection algorithm is to learn the features of new classes by only a small amount of labeled data of new classes, and to have the ability to detect new class targets in the detection task.

[0004] At present, there are two technical routes for small sample target detection. One is the meta-learning based method: the core idea of small sample learning based on meta-learning is to let the network learn how to learn, and the disadvantage of meta-learning is to rely on complex scene training. The second is the fine-tuning based method represented by TFA and DeFRCN: first, use a large public dataset as a base class to train the feature extraction ability of the model, then freeze the backbone network, and fine-tune the model using a small amount of new class data, to realize the small sample target detection ability on new classes. However, this method has the problems of contradictory optimization goals of subtasks, and classification errors caused by small amount of new class samples.

[0005] The present application uses a fine-tuning small sample target detection method, which aims to let the detection model learn the basic image features on a large amount of labeled public dataset, and in the fine-tuning stage, the model only learns the features of the target class to be detected through a small amount of target picture samples, and has the ability to detect the target to be detected in target detection. SUMMARY

[0006] The application aims at the problems of large training data demand, high data acquisition cost, poor performance of new class target detection, etc. in many automation application occasions, and provides a target detection method based on small samples. This method can reduce the dependence of target detection algorithm on a large amount of training data, alleviate the problems of difficult data acquisition and high cost of manually labeled data, relieve the contradiction between the mutual influence of different modules and the performance reduction, improve the detection performance without increasing the training cost, reduce the detection result errors caused by classification errors, and improve the classification performance of the model.

[0007] The technical solution for achieving the object of the application is:

[0008] A target detection method based on small samples, comprising the following steps:

[0009] S1: image acquisition: in a real environment, using an image acquisition device to take pictures of the target to be measured, and acquiring images of the target to be measured;

[0010] S2: image preprocessing: converting the format of the acquired images into JPG format, and uniformly setting the image size to 1920x1280 pixels, using LabelImg labeling software to label the target in each image, constructing a small sample target detection data set, the data set follows the labeling format and directory structure of Pascal VOC, and the data set is divided and a configuration file is generated according to the division method of small sample target detection;

[0011] S3: constructing a small sample target detection model, taking Faster R-CNN as the basic framework of small sample target detection, the framework mainly consists of a backbone feature extraction network, an RPN network and a detection network, and a gradient decoupling module and a contrast coding module are added on the basis of the framework;

[0012] The backbone feature extraction network is composed of a backbone network and a FPN (feature pyramid network). The backbone network adopts a residual structure ResNet 101, and contains a 7x7 convolutional layer Conv1 and four residual structures Conv2_x, Conv3_x, Conv4_x and Conv5_x. The four residual structures respectively output four feature maps CM2, CM3, CM4 and CM5. The FPN uses a 1x1 convolutional layer to adjust the channel number of the input feature map, and then fuses the feature maps CM2, CM3, CM4 and CM5 through upsampling to obtain fused feature maps M2, M3, M4 and M5. The fused feature maps are respectively subjected to a 3x3 convolutional layer to obtain feature maps P2, P3, P4 and P5, and a Maxpool operation is performed on the basis of P5 to obtain a feature map P6 through downsampling; the RPN network is used to predict candidate regions on the feature maps P2, P3, P4, P5 and P6, and map the candidate regions to the feature maps P2, P3, P4 and P5 as inputs of the subsequent network;

[0013] The detection network includes an ROI pooling layer, an ROI feature extractor, a classifier and a regressor;

[0014] The ROI pooling layer takes the candidate regions generated by the RPN network as inputs, and performs an ROI pooling operation on the features P2, P3, P4, P5 and P6 output by the backbone feature extraction network to generate candidate region features of different scales and unify the sizes of the features to 7x7 for feature output;

[0015] The ROI feature extractor is composed of a full connection layer, and is responsible for extracting overall feature information, fusing global features and candidate region features, and outputting to the classifier and the regressor for detection;

[0016] The classifier is responsible for determining the class information of the target in the target candidate frame. The classification loss is a cross-entropy loss, as shown below:

[0017] L cls class = -log p cls u

[0018] Wherein, u is the true label of the target, p is the softmax probability distribution predicted by the classifier p = p (p0,..., pu), and pu represents the probability of the classifier predicting that the current candidate region is of the class u; k u

[0019] The regressor is responsible for adjusting the position of the target candidate frame. The regression loss is a Smoothed-L1 loss, as shown below:

[0020] ​​​

[0021]

[0022] where t u is the bounding box regression parameter of the corresponding class u predicted by the regressor, v is the bounding box regression parameter of the real target, is the Smoothed-L1 function;

[0023] Gradient decoupling module: the role of this module is to adjust the coupling degree between different modules, to alleviate the problem of non-optimal solution of a single task caused by the inconsistent optimization objectives of sub-modules. The gradient decoupling module is used between the backbone feature extraction network and the RPN network, and between the backbone feature extraction network and the detection network, to adjust the coupling degree between the backbone feature extraction network and the RPN network and the detection network, and to alleviate the problem that the RPN network and the detection network are mutually influenced due to sharing the backbone network, resulting in performance reduction;

[0024] In the forward propagation process, the decoupling module performs affine transformation A(x) to map the feature map from the backbone network to a new feature space. The affine transformation is parameterized by trainable channel weights ω and bias b. In the backward propagation process, the decoupling module multiplies the backward gradient by a constant λ, thereby adjusting the gradient weight of the backward propagation. The forward propagation and the backward propagation are represented by the following formulas:

[0025] D (A，λ) (x)=A(x),

[0026]

[0027] where D (A，λ) represents gradient decoupling, A(x) is an affine transformation operation, λ∈[0,1] is a decoupling coefficient, L d is the loss downstream of the decoupling module, i.e., the loss of the RPN network or the loss of the detection network;

[0028] Contrastive encoding module: the contrastive encoding module draws on the method of contrastive learning to reduce the intra-class difference, increase the inter-class difference, and enhance the classification performance of the model. In this module, a classifier based on the cosine similarity function is used. The contrastive encoding module is parallel to the classifier and the regressor and is inserted behind the ROI feature extractor to convert the 1x1024 features into 1x128 features. The similarity of the features is calculated and a contrastive loss function is added to increase the identity of the same class proposal and the distinction between different classes.

[0029] S4: The small sample target detection model is trained in two stages using the base class dataset and the new class dataset respectively:

[0030] S4-1: In the first stage, Pascal VOC is used as the base class dataset to train the general feature extraction capability of the network, that is, the stochastic gradient descent is used as the optimization function, the standard batch size is 16, the momentum is 0.9, the weight decay is 0.0001, the initial learning rate is set to 0.02, for the decoupling coefficient λ of the gradient decoupling module, λ used for the RPN network is set to 0, and λ used for the detection network is set to 0.75; only the backbone feature extraction network, the gradient decoupling module, the RPN network and the detection network of the model are trained in this stage;

[0031] S4-2: In the second stage, the small sample target detection dataset that needs to be trained is used as a new class, and the same number of samples are sampled for all classes of the base class and new class dataset as training data, and the model is fine-tuned. The training samples are randomly sampled according to 5 samples, 10 samples, 15 samples and 30 samples to obtain a small sample training set;

[0032] In this stage, the model parameters obtained in step S4-1 are first loaded, and then the parameters of the backbone network ResNet 101 and the ROI pooling layer are frozen and do not participate in fine-tuning training, while the feature pyramid network, the RPN network, the contrast encoding module, the contrast encoding module, and the detection network excluding the ROI pooling layer are jointly fine-tuned and trained, and the learning rate adopts the MultiStepLR strategy. The learning rate is adjusted according to the set step interval during training, and the initial learning rate is set to 0.001.

[0033] S5: Test using the network model obtained in step S4: input the target image to be detected into the network model obtained in step S4, and output the class, positioning bounding box and confidence of the detected target from the model.

[0034] The technical solution method improves the Faster R-CNN as the basic framework. By adding the gradient decoupling module, the contradiction between different sub-modules is relieved; by the contrast encoding method, the performance of the model classification is enhanced; the two-stage fine-tuning small sample target detection method is used in the technical solution, and better performance is obtained by using a simpler training method. Compared with the existing technology, the technical solution has the following advantages:

[0035] (1) The small sample training method is used to reduce the dependence of the target detection algorithm on a large amount of training data, and to alleviate the problem of high data acquisition cost;

[0036] (2) The gradient decoupling module adjusts the coupling degree between the backbone feature extraction network, the RPN network and the detection network, and relieves the contradiction between different modules that affect each other and reduce the performance, thereby improving the detection performance without increasing the training cost;

[0037] (3) The method of contrast coding is adopted to reduce the detection result error caused by classification error and improve the classification performance of the model. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 Flowchart of the embodiment;

[0039] Figure 2 Working schematic diagram of the small sample based target detection model in the embodiment;

[0040] Figure 3 Schematic diagram of the gradient decoupling module;

[0041] Figure 4 Detection effect diagram of the embodiment on the citrus;

[0042] Figure 5 Detection effect diagram of the embodiment on the passion fruit;

[0043] Figure 6 Detection effect diagram of the embodiment on the apple. DETAILED DESCRIPTION

[0044] The content of the present application will be further described below in combination with the drawings and embodiments, but is not a limitation on the present application.

[0045] Embodiment:

[0046] The present example takes the orchard fruit detection of citrus, passion fruit and the like as an example.

[0047] Reference Figure 1 A small sample based target detection method, comprising the following steps:

[0048] S1: image acquisition: in the orchard environment, the user uses a camera or a mobile phone to take pictures of the fruits in the orchard, and obtains images of the fruits in the orchard;

[0049] S2: image preprocessing, constructing a small sample fruit target detection data set: using Photoshop to uniformly crop the picture to 1920x1280 pixels, converting the format to JPG format, using LabelImg labeling software to label the fruit target in each picture, constructing a small sample fruit target detection data set, the data set follows the annotation format and directory structure of Pascal VOC, the annotated XML file is placed in the Annotations folder, the image JPG file is placed in the JPEGImages, according to the division method of the small sample target detection, according to different sample quantities, using random sampling to sample the samples, dividing the new class data set in the fine tuning stage, and generating the configuration file of data division;

[0050] S3: Construct a small sample target detection model for orchard fruit detection, which is a two-stage fine-tuning method. Faster R-CNN is used as the basic framework for small sample target detection, which mainly consists of a backbone feature extraction network, an RPN network, and a detection network. On this basis, a gradient decoupling module and a contrast encoding module are added:

[0051] The backbone feature extraction network is composed of a backbone network and a feature pyramid network. The backbone network uses ResNet101 with a residual structure, which includes a 7x7 convolutional layer Conv1 and four residual structures Conv2_x, Conv3_x, Conv4_x, and Conv5_x. After receiving the input fruit image, the four residual structures output four feature maps CM2, CM3, CM4, and CM5, respectively. FPN uses a 1x1 convolutional layer to adjust the channel number of the input feature map, and then uses upsampling to fuse the feature maps CM2, CM3, CM4, and CM5 to obtain the fused feature maps M2, M3, M4, and M5. After the fused feature maps pass through a 3x3 convolutional layer, they obtain feature maps P2, P3, P4, and P5, and use Maxpool operation on P5 to downsample the feature map P6;

[0052] The RPN network is used to generate candidate regions and map them to feature maps as input for the subsequent network. Specifically, the RPN network receives different scale features P2, P3, P4, P5, and P6 from the backbone feature extraction network as input, uses a sliding window to generate multiple candidate regions of different sizes and scales, performs candidate region prediction, and finally outputs a set of candidate regions with higher scores and more accurate positions, which are mapped to features P2, P3, P4, and P5 of the feature pyramid network;

[0053] The detection network includes a ROI pooling layer, a ROI feature extractor, a classifier, and a regressor.

[0054] The ROI pooling layer takes the candidate regions generated by the RPN network as input, performs ROI pooling operation on the features P2, P3, P4, P5, and P6 output by the backbone feature extraction network, generates candidate region features of different scales, and outputs them as features of size 7x7.

[0055] The ROI feature extractor is composed of fully connected layers, which are responsible for extracting overall feature information, fusing global features and candidate region features, and outputting to the classifier and regressor for detection.

[0056] The classifier is responsible for determining the class information of the fruit in the target candidate box. The classification loss is the cross-entropy loss, which is shown as follows:

[0057] Lcls = L cls (p, i) = -log p u

[0058] where u is the true label of the target, p is the softmax probability distribution predicted by the classifier p = p(p0,..., pu,..., pK), and p k represents the probability of the classifier predicting the current candidate region as class u; u

[0059] The regressor is responsible for adjusting the position of the target candidate box, and the regression loss is the Smoothed-L1 loss, as follows:

[0060]

[0061]

[0062] where t u is the bounding box regression parameter of the corresponding class u predicted by the regressor, v is the bounding box regression parameter of the real target, and is the Smoothed-L1 function;

[0063] Gradient decoupling module: the role of this module is to adjust the coupling degree between different modules, and to alleviate the problem of non-optimal solution of a single task caused by the inconsistent optimization objectives of sub-modules, as shown in Figure 2 The gradient decoupling module is used between the backbone feature extraction network and the RPN network, and between the backbone feature extraction network and the detection network, to adjust the coupling degree between the backbone feature extraction network and the RPN network and the detection network, and to alleviate the problem of mutual influence between different modules caused by the shared backbone network of the RPN network and the detection network, resulting in performance degradation;

[0064] As shown in Figure 3 , in the forward propagation process, the decoupling module performs affine transformation A(x) to map the feature map from the backbone network to a new feature space, and the affine transformation is parameterized by trainable channel weights ω and bias b; in the backward propagation process, the decoupling module multiplies the backward gradient by a constant λ, thereby adjusting the gradient weight of the backward propagation; the forward propagation and the backward propagation can be represented by the following formula:

[0065] D (A，λ )(x) = A(x),

[0066]

[0067] where D (A，λ) represents gradient decoupling, A(x) is an affine transformation operation, λ ∈ [0, 1] is a decoupling coefficient, and L d ​is the loss downstream of the decoupling module, i.e., the loss of the RPN network or the loss of the detection network;

[0068] Contrastive encoding module: the contrastive encoding module draws on the method of contrastive learning to reduce the intra-class difference, increase the inter-class difference, and enhance the model classification performance. The module adopts a classifier based on a cosine similarity function, as shown in the following formula: Figure 2 The contrastive encoding module is parallel to the classifier and the regressor and is inserted behind the ROI feature extractor to convert the 1x1024 features into 1x128 features, calculate the similarity of the features, and add a contrastive loss function to increase the identity of the same class proposal and the distinction between different classes.

[0069] S4: The small sample target detection model is trained in two stages using the base class dataset and the new class dataset respectively. In this example, the training graphics card is NVIDIA GeForce RTX 3090, and the processor is Intel(R) Xeon(R) Silver4114 CPU @ 2.20GHz. The software environment for training is Ubuntu 20.04, CUDA Version: 11.4, Pytorch 1.9, and Python 3.9.

[0070] S4-1: In the first stage, Pascal VOC is used as the base class dataset to train the general feature extraction capability of the network, i.e., using stochastic gradient descent as the optimization function, the standard batch size is 16, the momentum is 0.9, the weight decay is 0.0001, the initial learning rate is set to 0.02, the decoupling coefficient λ of the gradient decoupling module is set to 0 for the RPN network and 0.75 for the detection network. In this stage, only the backbone feature extraction network, the gradient decoupling module, the RPN network, and the detection network of the model are trained.

[0071] S4-2: In the second stage, the small sample fruit target detection dataset constructed is used as a new class. For all classes of the base class and the new class dataset, the same number of samples are sampled as training data, and the model is fine-tuned. The training samples are randomly sampled according to 5 samples, 10 samples, 15 samples, and 30 samples to obtain the small sample training set.

[0072] In this stage, the model parameters obtained in step S4-1 are first loaded, and then the parameters of the backbone network ResNet 101 and the ROI pooling layer are frozen and do not participate in the fine-tuning training, while the feature pyramid network, the RPN network, the contrast encoding module, the contrast encoding module, and the detection network without the ROI pooling layer are jointly fine-tuned, and the learning rate adopts the MultiStepLR strategy, that is, the learning rate is adjusted according to the set step interval during training, the initial learning rate is set to 0.001, the decoupling coefficient λ used for the RPN network is set to 0, and the decoupling coefficient λ used for the detection network is set to 0.001;

[0073] S5: Test the network model obtained in step S4 using orchard fruit images: use the command line to input the fruit image file name to be detected and the network model obtained in step S4, and output the class, positioning bounding box and confidence of the detection target in the fruit image.

[0074] In this example, the images of oranges, passion fruits and apples collected in the orchard environment are input into the network model obtained in step S4 for target detection, and the detection results are as shown in Figure 4 、 Figure 5 and Figure 6 . The detection results of the orchard fruit dataset in this example are shown in Table 1. The detection results show that this example can realize the demand of orchard fruit target detection under the condition of only a small amount of training samples.

[0075] Table 1: Performance of this example in fruit target detection using AP50 as an evaluation index, where AP50 represents mAP>0.5.

[0076]

[0077] The method of this example can also be applied to items with similar characteristics to tangerines, apples and passion fruits.

Claims

1. A small sample based target detection method, characterized in that, Comprising the following steps: S1: image acquisition: in a real environment, using an image acquisition device to take a real scene of the target to be measured, and obtaining an image of the target to be measured; S2: image preprocessing: converting the format of the obtained image into JPG format, and uniformly setting the image size to 1920x1280 pixels, using LabelImg labeling software to label the target in each image, constructing a small sample target detection data set, the data set follows the annotation format and directory structure of Pascal VOC, and according to the division method of small sample target detection, different sample quantities are adopted, and the new class data set of the fine-tuning stage is divided by random sampling, and the configuration file of data division is generated; S3: constructing a small sample target detection model, taking Faster R-CNN as the basic framework of small sample target detection, which mainly consists of a backbone feature extraction network, an RPN network and a detection network, and adding a gradient decoupling module and a contrast coding module based on the framework; The backbone feature extraction network is composed of a backbone network and an FPN, the backbone network adopts a residual structure ResNet 101, including a 7x7 convolution layer Conv1 and four residual structures Conv2_x, Conv3_x, Conv4_x and Conv5_x, the four residual structures output four feature maps CM2, CM3, CM4 and CM5 respectively, the FPN uses a 1x1 convolution layer to adjust the channel number of the input feature map, and then performs upsampling on the feature maps CM2, CM3, CM4 and CM5 to obtain fused feature maps M2, M3, M4 and M5, and the fused feature maps are respectively output after passing through a 3x3 convolution layer to obtain feature maps P2, P3, P4 and P5, and a Maxpool operation is performed on the basis of P5 to obtain a feature map P6; The RPN network is used to predict candidate regions on the feature maps P2, P3, P4, P5 and P6, and map the candidate regions to the feature maps P2, P3, P4 and P5 as inputs for the subsequent network; The detection network includes an ROI pooling layer, an ROI feature extractor, a classifier and a regressor; The ROI pooling layer takes the candidate regions generated by the RPN network as input, performs ROI pooling operation on the features P2, P3, P4, P5 and P6 output by the backbone feature extraction network, generates candidate region features of different scales and outputs the features with a size of 7x7; The ROI feature extractor is composed of a full connection layer, which is responsible for extracting overall feature information, fusing global feature and candidate region feature, and outputting to the classifier and regressor for detection; The classifier is responsible for determining the class information of the target in the target candidate box, and the classification loss is cross entropy loss, as shown below: ； wherein, the true label targeted, the softmax probability distribution predicted by the classifier , denotes the probability that the classifier predicts the current candidate region to be of class . The regressor is responsible for adjusting the position of the target candidate box, and the regression loss is Smoothed-L1 loss, as shown below: ； ， wherein, are the bounding box regression parameters of the corresponding class u predicted by the regressor, and v are the bounding box regression parameters of the real target, is the Smoothed-L1 function; Gradient decoupling module: the role of this module is to adjust the coupling degree between different modules, to alleviate the problem of non-optimal solution of a single task caused by inconsistent optimization objectives of sub-modules, the gradient decoupling module is used between the backbone feature extraction network and the RPN network, and between the backbone feature extraction network and the detection network, to adjust the coupling degree between the backbone feature extraction network and the RPN network and the detection network, and to alleviate the problem that the RPN network and the detection network are mutually influenced due to sharing the backbone network, resulting in performance reduction; In the forward propagation process, the decoupling module performs affine transformation A(x) to map the feature map from the backbone network to a new feature space, and the affine transformation is parameterized by trainable channel weights ω and bias b. In the backward propagation process, the decoupling module multiplies the backward gradient by a constant λ, thereby adjusting the gradient weight of the backward propagation. The forward propagation and the backward propagation are represented by the following formulas: ， ， wherein, denotes gradient decoupling, is an affine transformation operation, is a decoupling coefficient, is a loss downstream of the decoupling module, i.e. a loss of the RPN network or a loss of the detection network; Contrastive coding module: the contrastive coding module learns from the method of contrastive learning, to reduce the intra-class difference, increase the inter-class difference, and enhance the classification performance of the model. In this module, a classifier based on cosine similarity function is used. The contrastive coding module is parallel to the classifier and the regressor, and is inserted behind the ROI feature extractor to convert the 1x1024 feature into 1x128 feature. The similarity of the features is calculated and a contrastive loss function is added to increase the identity of the same class proposal and the distinction between different classes. S4: Two-stage training of the small sample target detection model using the base class dataset and the new class dataset respectively: S4-1: In the first stage, Pascal VOC is used as the base class dataset to train the general feature extraction capability of the network, that is, the random gradient descent is used as the optimization function, the standard batch size is 16, the momentum is 0.9, the weight decay is 0.0001, the initial learning rate is set to 0.02, and the decoupling coefficient of the gradient decoupling module is 0 for the RPN network 0 for the detection network 0.75 is set; only the backbone feature extraction network, the gradient decoupling module, the RPN network and the detection network of the model are trained in this stage; S4-2: In the second stage, the small sample target detection dataset to be trained is used as a new class. For all classes of the base class and new class datasets, the same number of samples are sampled as training data, and the model is fine-tuned. The training samples are randomly sampled according to 5 samples, 10 samples, 15 samples and 30 samples to obtain the small sample training set; In this stage, the model parameters obtained in step S4-1 are first loaded, and then the parameters of the backbone network ResNet 101 and the ROIpooling layer are frozen and do not participate in fine-tuning training. At the same time, the FPN, RPN network, contrastive coding module, contrastive coding module, and detection network without ROI pooling layer are jointly fine-tuned and trained. The learning rate adopts the MultiStepLR strategy, and the learning rate is adjusted according to the set step interval during training. The initial learning rate is set to 0.

001. S5: Test the network model obtained in step S4: input the target image to be detected into the network model obtained in step S4, and output the class, positioning bounding box and confidence of the detected target from the model.

Images