Small-sample target detection method based on graph convolution and meta learning

Abstract

The invention provides a few-sample target detection method based on graph convolution and meta learning. The method comprises the following steps of: firstly, sampling a support set by using a certain ROI (Region of Interest) feature in a query set to obtain a support set prototype feature which is the same as the query set in category; and secondly, using a support category label obtained by sampling and a query set ROI soft classification label to carry out graph convolutional neural network forward conduction, and finally calculating cross entropy loss by using an obtained output graph classification result and an original label, so as to guide classification and relieve the problem of'domain offset '. And finally, carrying out graph convolutional neural network forward conduction by using all support category labels and sampled query ROI soft classification label composition, and finally calculating cross entropy loss by using another obtained graph classification result and an original label, so as to guide element loss and relieve the problem of forgetting of the model. The method is mainly applied to a video monitoring intelligent analysis application system, and has a wide application prospect in the fields of industrial vision, intelligent security, military and the like.

Description

Technical Field

[0001] The present invention relates to a few-sample target detection method based on graph convolution and meta-learning, and designs a few-sample target detection framework with a new classification loss and meta-loss network, which relates to the problem of few-sample target detection in the field of video intelligent monitoring and belongs to the field of computer vision and intelligent information processing. Background Art

[0002] Object detection is an important branch in the field of computer vision, which includes two tasks, namely classification and positioning. Early deep learning-based object detection tasks are usually carried out in multiple stages. For example, the Faster-RCNN network first determines the foreground and background in the image, and then further classifies and regresses the foreground object. Compared with multi-stage detectors, the more popular one-stage YOLO series of object detectors do not need to predict the foreground and background first, but directly classify and regress the target object, thus having a significant advantage in speed compared to multi-stage object detectors. With the development of object detection technology, it has been widely used in many fields, including cross-domain object detection, semi-supervised object detection, directed object detection, and few-sample object detection.

[0003] Few Shot Object Detection (FSOD) is a new direction in the field of object detection in recent years, which aims to train a robust model with limited labeled samples. As we all know, deep learning is data-driven, and it is easy to overfit in limited labeled scenarios; however, data is not easy to obtain in many fields, such as some confidential data in remote sensing scenarios, some prohibited items in security scenarios, and some expensive instruments in industrial vision fields; in addition, a large amount of data annotation will also consume a lot of manpower and material resources; therefore, few-shot object detection provides a solution for the above scenarios. However, images in real scenes will have many problems due to imaging mechanisms, such as low image contrast in X-ray imaging scenarios and multi-scale targets in remote sensing scenarios, which aggravates the difficulty of few-shot object detection.

[0004] For the few-shot target detection technology, there are four main paradigms today: meta-learning, data augmentation, metric learning, and transfer learning. Among them, the meta-learning paradigm combines the meta-learning algorithm to perform few-shot target detection, which requires the division of meta-tasks. There are 200 meta-tasks in the base class training phase and they are only composed of base classes. There are only a limited number of meta-tasks in the new class training phase and they include base classes and new classes. This method can quickly adapt to new tasks, but the overall detection accuracy is not high; based on the data augmentation paradigm, data augmentation is performed on limited labeled data, such as using an illusioner for data augmentation, and using rotation data augmentation in remote sensing scenarios. This method can solve the few-shot target detection task in scenarios with extremely scarce data, such as only one instance annotation; based on the metric learning method, the support data (Support Set) and the query data (Query Set) are classified by measuring the similarity distance between them, and the target position is calculated through RPN (Region Proposal Network) and ROI (Region Of Interest) is determined, and the global, local, and patch relationships are usually used to measure foreground and background targets, allowing the network to further learn the relationship between target objects. This paradigm only needs to be trained once, but the accuracy is still limited; the transfer learning paradigm is the most widely used method. By pre-training the base class data (containing a large number of annotations), and then fine-tuning on the new class (few sample annotations) data, few-sample target detection can be achieved. This paradigm can achieve good detection accuracy, but the generalization is still limited in the face of complex scenarios.

[0005] Based on the meta-learning paradigm but different from this method, the present invention draws on the advantage of graph convolutional neural networks in being able to handle relationships between categories well, and performs relationship learning by mapping category information to a high-dimensional non-Euclidean space. Specifically, the "S1-Qn" method is proposed, which maps the sampled support category labels and the soft classification labels of all query category ROIs to a high-dimensional space for category relationship learning and redesigns the classification loss. The learned category relationship information is used to guide the few-sample target detection to better classify and alleviate the "domain shift" problem. Similarly, the "Sn-Q1" method is proposed, which maps all support category labels and the soft classification labels of the query category ROI that participates in the sampling support set to a high-dimensional space for category relationship learning and redesigns the meta-loss based on this process. The learned category relationship information is used to guide the meta-task and alleviate the "forgetting" problem. Summary of the invention

[0006] The present invention proposes a few-shot target detection method based on graph convolution and meta-learning, and designs a new few-shot target detection framework with improved classification loss and improved meta-learning loss, including the following steps:

[0007] (1) Extract features from the support set and reduce the category features to two dimensions. Then, calculate the template of the corresponding category for all the two-dimensional features, that is, take the average operation of all the two-dimensional features under the corresponding category.

[0009] (2) Extract features for the query category and extract ROI using RPN. Similarly, the query ROI features are also reduced in dimension

[0010] Into two dimensions;

[0011] (3) Randomly select a ROI category in the query graph to sample the supporting prototype category, and obtain the same supporting prototype category as the query ROI category;

[0012] (4) Perform forward inference on the “S1-Qn” and “Sn-Q1” algorithms, calculate losses, and update parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is an overall block diagram of a few-sample target detection method based on graph convolution and meta-learning in the present invention;

[0014] Figure 2 It is a structural diagram of the "S1-Qn" module of the present invention;

[0015] Figure 3 This is a structural diagram of the "Sn-Q1" module of the present invention. DETAILED DESCRIPTION

[0016] The following is combined with Figure 1 , Attachment Figure 2 , attached Figure 3 The present invention will be further described:

[0017] Figure 1 Shown is the overall network structure and principle of the few-shot target detection algorithm based on meta-learning and graph convolutional neural network:

[0018] The few-shot object detection framework based on meta-learning and graph convolutional neural network is trained and tested in an end-to-end manner. The network mainly consists of four parts: dual-stream input, feature extraction, support category sampling, GCN forward reasoning and loss calculation. In the dual-stream input part, the upper branch inputs the query graph, which needs to be extracted by the RPN network. The lower branch inputs the support category, following the N-way-K-shot paradigm. The query graph in the upper branch is reduced to a two-dimensional feature vector after extracting the ROI features by the RPN. The lower branch annotates the N categories and the corresponding K support instances of each category, and performs category template calculation, that is, feature extraction is performed on all categories and all support instances of each category, and then the average operation is taken to obtain the corresponding class prototype. The sampling process first randomly selects a ROI in the query graph and obtains its category. Using the query ROI, the prototype with the same category in the support prototype is found to complete the sampling process of the support prototype. The sampled support category labels, all query ROI feature soft classification labels, all support prototype labels, and a query ROI soft classification label participating in the sampled support prototype are mapped to the high-dimensional space for GCN forward propagation, the graph loss is calculated, and the improved classification loss and meta-loss are calculated, and finally the network parameters are updated.

[0019] (1) Support category prototype calculation and aggregation operations

[0020] The meta-task design in the present invention is based on the N-way-K-shot paradigm. For the supporting categories, the features of each instance are first extracted, and then the corresponding prototype is calculated. The calculation method is shown in formula (1):

[0021]

[0022] where f i pro is the prototype of the ith category, avg(·) is the arithmetic average operation, and f i t is the label of the tth instance of the i-th category, and K is the total number of shots. For the query image of the upper branch, after the CNN network extracts features, the RPN network generates a series of suggestion boxes. 512 query features are obtained in one episode (batch size is 4), and each batch is 128; then they are reshaped into two-dimensional vectors; then the support graph sampling operation is performed. The specific process is to randomly select a query vector from the query graph and obtain its category information, and then obtain the support graph feature vector with the same category as the query image from the prototype library, and perform a dot product operation on the sampled support graph feature vector and the previous 128 query graph feature vectors to obtain the aggregate feature (Aggregate Feature), as shown in formula (2):

[0023]

[0024] Among them, f t agg is the tth aggregate feature, is the prototype obtained by sampling, f t q is the t-th query feature, and ⊙ is the dot product operation. The aggregated features are finally sent to two fully connected layers for regression and classification operations and loss calculation.

[0025] (2) “S1-Qn” method

[0026] Figure 2 The "S1-Qn" method proposed in the present invention is shown. In each meta-task of meta-learning, the support set needs to be sampled using the ROI category in the query set, that is, first clarify which query feature to use, and then find the corresponding support feature through the query feature. Therefore, the feature sampled in this process determines the target category of the meta-task. S1-Qn is to form a one-to-many relationship between the sampled support feature category label and the soft classification results of the 128 query set ROIs, map them to a high-dimensional space, and use the sampled support prototype to guide the query category to perform high-dimensional relationship learning. The network maps the sampled support set labels and the softened query ROI classification confidence as graph nodes to a non-Euclidean space to establish a graph structure. The temperature function is shown in formula (3):

[0027]

[0028] Among them, l i is the i-th softened query set ROI category classification label, and T is the temperature hyperparameter. roi is the sum of the number of query set ROIs. The sampled support set features participate in the covariance calculation together with the query set ROI features, and the result is used as the adjacency matrix of the graph and input into the GCN network. Since GCN has requirements for the number of hidden layers, too deep a network depth will lead to excessive smoothing of node outputs. Therefore, the number of hidden layers is set to ensure the best GCN output results. Finally, the output result is sent to the Softmax function for classification, and the original sampled support labels and the labels corresponding to the query set ROI are concatenated. After concatenation, the GCN loss is calculated, as shown in equations (4)-(7):

[0029] l gt =Concat(l support_sample, l query_roi ) (4)

[0030] p=softmax(F cls ([f support_sample ,f query ])) (5)

[0031]

[0032] L cls_all =L cls ++L gcn_cls (7)

[0033] Among them, l support_sample is the original sampled support feature label, l query_roi are the labels of all query ROIs, Concat(·) is the concatenation operation, l gt is the concatenated label. support_sample is the sampled support feature, f query is the ROI feature of all query sets, F cls (·) is the classification operation through the classifier, softmax(·) is the normalized exponential function, p is the final overall prediction result after GCN, and L cls is the classification loss. L gcn_cls It is the graph convolution loss in the "S1-Qn" case. This loss is finally included in the overall classification loss of the network and participates in the forward propagation together, compensating the network for learning the relationship between query categories in non-Euclidean space, thereby improving the classification recognition ability. cls_all It is the improved detector classification loss function in the present invention.

[0034] (3) “Sn-Q1” method

[0035] In response to the "forgetting" problem in the field of few-sample target detection, the present invention proposes a "Sn-Q1" method, that is, all category labels of the support set and the sampled query ROI feature soft classification confidences are simultaneously mapped to a non-Euclidean space to construct a graph structure, and then the correlation between the support set categories is learned through a graph convolutional neural network. Since the meta-task of the meta-learning algorithm samples a category as a positive sample, and then the query set ROI is feature aggregated through the positive sample support feature, and "Sn-Q1" only maps a sampled query ROI to the embedding space for dynamic learning, as long as the graph convolution loss in the "Sn-Q1" scenario is added to the meta-classification loss to guide the meta-learning parameter update, the meta-parameters can be included in the sub-task relationship information learned in the high-dimensional space, thereby optimizing the meta-learning algorithm. Figure 3It can be seen that the overall structure of the network is similar to that of the "S1-Qn" scenario. The difference is that the network only sends the sampled query feature to the temperature function for classification operation. After obtaining the corresponding score, it is used together with all the support set category labels as nodes. Similarly, the covariance matrix composed of the covariance coefficients between the features of all prototypes in the support set and the ROI features of the sampled query set is calculated as the adjacency matrix of the high-dimensional space graph. After passing through multiple hidden layers of GCN, it is classified by the fully connected layer. The score vector obtained after classification is then cross-entropy calculated with the concatenated original support set label and the sampled query label to calculate the graph loss, as shown in equations (8)-(11):

[0036] l' gt =Concat(l support ,l query_sample ) (8)

[0037] p'=softmax(F cls ([f support ,f query_sample ])) (9)

[0038]

[0039] L meta_cls_all =L meta_cls +L gcn_meta_cls (11)

[0040] Among them, l' gt is the concatenation of all supporting prototype labels under “Sn-Q1” and the sampled query set category labels, f support are all support set prototype features, f query_sample is the sampled query set feature, F cls (·) is the classification operation, softmax(·) is the normalized exponential function, and p' is the graph classification score vector through the GCN network. Yes gt The i-th component in i ' is the i-th component in p', L gcn_meta_cls is the graph convolution loss, L meta_cls is the meta-classification loss, L meta_cls_all It is the improved overall meta-classification loss of the present invention. Directly adding the graph loss to the original meta-classification loss can assist the meta-learner in learning the correlation information of each meta-task in the high-dimensional space, thereby alleviating the "forgetting" problem.

[0041] The present invention conducts a network structure ablation study on the VOC0712 dataset, where Baseline represents the baseline network, "S1-Qn" represents the "S1-Qn" method proposed by the present invention, and "Sn-Q1" represents the "Sn-Q1" method proposed by the present invention. In addition, it is compared with some mainstream algorithms on the VOC0712 dataset and the MS-COCO dataset, using nAP as the evaluation index. The experimental results are shown in Table 1, Table 2, and Table 3. The experimental accuracy is significantly improved compared with the baseline network and other comparison algorithms.

[0042] Table 1 Ablation test mAP (%) of all categories in each module

[0043]

[0044] Table 1

[0045]

[0046] Table 2 Test results of MS-COCO dataset nAP (%)

[0047]

[0048] Table 3 Chapter 3 VOC0712 dataset test results nAP (%)

[0049]

Claims

1. A few-shot object detection method based on graph convolution and meta-learning, It is characterized by the following steps: (1) First, the query graph and support graph are input into the Siamese structure network for feature extraction. The query graph is extracted through the upper branch and RPN for ROI extraction. The support graph follows N-way-K-shot and completes the prototype calculation through the lower branch. A ROI category in the query graph is randomly selected to sample the support set. (2) Map the sampled support category labels and all query ROI feature soft classification labels together into a non-Euclidean space for category relationship learning, and use this relationship information to redesign the classification loss to guide network classification; (3) The query feature ROI soft classification labels of the sampling support categories and all support category labels are mapped to the non-Euclidean space for category relationship learning, and the learned relationship information is used to redesign the meta-loss and guide the meta-parameter update.

2. According to the method of claim 1, the meta-task design of step (1) is based on the N-way-K-shot paradigm. For the supporting category, firstly, the features of each instance are extracted, and then the corresponding prototype is calculated. The calculation method is shown in formula (1): where f i pro is the prototype of the ith category, avg(·) is the arithmetic average operation, and f i t is the annotation of the tth instance of the ith category, and K is the total number of shots; for the query image of the upper branch, after the CNN network extracts features, the RPN network generates a series of suggestion boxes. 512 query features are obtained in one episode (batch size is 4), and each batch is 128; then it is also reshaped into a two-dimensional vector; then the support graph sampling operation is performed. The specific process is to randomly select a query vector from the query graph and obtain its category information, and then obtain the support graph feature vector with the same category as the query image from the prototype library, and perform a dot product operation on the sampled support graph feature vector and the previous 128 query graph feature vectors to obtain the aggregate feature (Aggregate Feature), as shown in formula (2): in, f t agg is the tth aggregate feature, is the prototype obtained by sampling, f t q is the t-th query feature, ⊙ is the dot product operation, and the aggregated features are finally sent to the two fully connected layers for regression classification operations and loss calculation.

3. According to the method of claim 1, the transformation alignment is that in step (2), the present invention uses the "S1-Qn" module, and Figure 2 shows the "S1-Qn" method proposed in the present invention; in each meta-task of meta-learning, the ROI category in the query set needs to be used to sample the support set, that is, first, it is clear which query feature to use, and then the corresponding support feature is found through the query feature, so the feature sampled in the process determines the target category of the meta-task; "S1-Qn" is to form a one-to-many relationship between the sampled support feature category label and the soft classification results of the 128 query set ROIs, map them to a high-dimensional space, and use the sampled support prototype to guide the query category to perform high-dimensional relationship learning; The network maps the sampled support set labels and the softened query ROI classification confidence as graph nodes to the non-Euclidean space to establish a graph structure. The temperature function is shown in formula (3): Among them, l i is the i-th softened query set ROI category classification label, T is the temperature hyperparameter, n roi is the sum of the number of query set ROIs; the sampled support set features participate in the covariance calculation together with the query set ROI features, and the result is used as the adjacency matrix of the graph and input into the GCN network. Since GCN has requirements for the number of hidden layers, too deep a network depth will lead to excessive smoothing of node outputs, so the number of hidden layers is set to ensure the best GCN output result; finally, the output result is sent to the Softmax function for classification, and the original sampled support labels and the labels corresponding to the query set ROI are concatenated. After concatenation, the GCN loss is calculated, as shown in equations (4)-(7): l gt =Concat(l support_sample, l query_roi ) (4) p=softmax(F cls ([f support_sample, f query ])) (5) L cls_all =L cls ++L gcn_cls (7) Among them, l support_sample is the original sampled support feature label, l query_roi are the labels of all query ROIs, Concat(·) is the concatenation operation, l gt is the concatenated label; f support_sample is the sampled support feature, f query is the ROI feature of all query sets, F cls (·) is the classification operation through the classifier, softmax(·) is the normalized exponential function, p is the final overall prediction result after GCN, and L cls is the classification loss; L gcn_cls It is the graph convolution loss in the "S1-Qn" case. This loss is finally included in the overall classification loss of the network and participates in the forward propagation together, compensating the network for learning the relationship between query categories in non-Euclidean space, thereby improving the classification recognition ability; L cls_all It is the improved detector classification loss function in the present invention.

4. The method according to claim 1, characterized in that In step (3), as can be seen from Figure 3, the overall structure of the network is similar to the "S1-Qn" scenario. The difference is that the network only sends the sampled query feature to the temperature function for classification operation, and after obtaining the corresponding score, it is used together with all the support set category labels as nodes. Similarly, the covariance matrix composed of the covariance coefficients between the features of all prototypes of the support set and the sampled query set ROI features is calculated as the adjacency matrix of the high-dimensional space graph. After passing through multiple hidden layers of GCN, it is classified by the fully connected layer. The score vector obtained after classification is then cross-entropy calculated with the concatenated original support set label and the sampled query label to calculate the graph loss, as shown in equations (8)-(11): l' gt =Concat(l support ,l query_sample ) (8) p'=softmax(F cls ([f support ,f query_sample ])) (9) L meta_cls_all =L meta_cls +L gcn_meta_cls (11) Among them, l' gt is the concatenation of all supporting prototype labels under "Sn-Q1" and the sampled query set category labels, f support are all support set prototype features, f query_sample is the sampled query set feature, F cls (·) is the classification operation, softmax(·) is the normalized exponential function, and p' is the graph classification score vector through the GCN network. Yes gt The i-th component in i ' is the i-th component in p', L gcn_meta_cls is the graph convolution loss, L meta_cls is the meta-classification loss, L meta_cls_all is the improved overall meta-classification loss of the present invention.

Images