Sample target detection method and apparatus

Abstract

The application provides a sample target detection method and device, wherein the method comprises: acquiring a sample to be detected; inputting the sample to be detected into a calibrated classification network to obtain a corresponding target category. The application enhances the work of RoI features in a few-sample target detection task by aggregating global and local context knowledge, avoids overfitting problems, and helps to learn more stable boundaries for foreground and background categories.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer technology, and particularly relates to a sample target detection method and device. BACKGROUND

[0002] Target detection is a fundamental problem in computer vision, aiming to localize and classify objects in a given image. It has been widely applied in many practical applications, such as robot navigation, intelligent video surveillance, and aerospace. In the past few years, researchers have proposed many novel one-stage and two-stage object detectors to perform the general object detection task and achieved great success. However, these detectors heavily rely on large-scale benchmark datasets with accurate bounding box annotations, which are labor-intensive. In some cases (e.g., high cost of manual data annotation or scarcity of samples of certain classes), it can be impractical to collect large amounts of labeled data. When only limited annotated bounding boxes are available, the performance of the detector will usually drop significantly. In contrast, humans can quickly and accurately identify objects in an image given only a few annotated samples, using their prior knowledge. To simulate the learning mode of humans and reduce the dependence on large-scale annotated benchmark datasets, researchers have begun to turn to the few-shot object detection problem: given only a few annotated samples of new classes during training, they try to learn an object detector that can correctly predict the location and class of objects / targets of that class during testing.

[0003] In order to adapt the state-of-the-art deep learning-based general detectors to scenarios with fewer annotated samples, the following two challenges must be addressed: (1) Since most existing deep learning-based detectors are implemented based on backbone networks with a large number of stacked convolutional layers (e.g., residual networks), a small number of samples of a new category is always insufficient to learn representative sample features. (2) Due to the low data diversity, it is difficult for classifiers or regressors to identify the true sample distribution space of new category objects with only a few samples. Therefore, the learned few-shot detectors usually suffer from model overfitting problems. To address the above challenges, researchers have proposed some effective methods: (1) For the first problem, recent methods focus on learning more representative RoI (region of interest) features to enhance the classification / localization process. These methods can be roughly summarized into two categories: using global context information (i.e., exploring the relational information of different categories in the entire dataset to enhance RoI features) and using local context information (i.e., only considering the relational clues of local regions in the query image). In the former, a popular solution is to use the annotation boxes of objects of different categories to generate prototypes as reweighting coefficients to improve the basic RoI features of new category objects / targets. In the latter, a typical approach is to apply an attention module between the RoI and its context to automatically integrate local relational information in the query image. However, none of these methods uniformly utilize global and local contextual information. (2) To deal with the problem of model overfitting, some existing methods attempt to transfer source domain meta-level knowledge from old classes with sufficient annotations to construct detectors for new classes with limited annotations in the target domain. An increasingly popular solution is to adopt a two-step training paradigm, first pre-training the target detector on old classes with sufficient data, and then fine-tuning it on a balanced training set consisting of old and new classes. However, these methods rely only on limited samples of new classes to train the target classifier and cannot learn the classification boundaries well. Summary of the Invention

[0004] The present invention provides a sample target detection method and device to solve the defects in the prior art that a small number of samples of a new category are always insufficient to learn representative sample features and there is overfitting, thereby alleviating the overfitting problem caused by the low diversity of data of the new category.

[0005] In a first aspect, the present invention provides a sample target detection method, comprising:

[0006] Obtain samples to be tested;

[0007] The sample to be detected is input into the calibrated classification network to obtain the corresponding target category.

[0008] Furthermore, according to a sample target detection method provided by the present invention, the calibrated classification network is trained by the following method:

[0009] obtain basic category data and new category data;

[0010] train an initial classification network according to the basic category data to obtain a trained classification network;

[0011] adjust the trained classification network according to the basic category data and the new category data to obtain the calibrated classification network.

[0012] Further, according to the sample target detection method provided by the present application, the training of the initial classification network according to the basic category data to obtain the trained classification network specifically comprises:

[0013] construct first support data and first query data according to the basic category data;

[0014] obtain a prototype of a first category according to the first support data;

[0015] construct a first category topology graph according to the prototype of the first category, and obtain first representative category feature knowledge according to the first category topology graph;

[0016] determine a conversion prototype of a first foreground category, a conversion prototype of a first proxy category, and a corresponding first meta-loss according to the first representative category feature knowledge;

[0017] obtain a first region division of the first query data and a corresponding first refinement loss;

[0018] determine a corresponding first initial ROI feature according to the first region division;

[0019] determine a first enhanced ROI feature according to the first initial ROI feature and the first representative category feature knowledge;

[0020] perform regression and classification on the first query data according to the first enhanced ROI feature to obtain a corresponding first regression loss and a first classification loss;

[0021] determine a first calibration loss according to trainable parameters of a basic frame classifier in the initial classification network, the conversion prototype of the first foreground category, and the conversion prototype of the first proxy category;

[0022] determine a first overall loss according to the first meta-loss, the first refinement loss, the first regression loss, the first classification loss, and the first calibration loss;

[0023] Adjusting parameters of the initial classification network to obtain a minimum first overall loss, taking the initial classification network corresponding to the minimum first overall loss as a trained classification network and fixing training parameters in the trained classification network except for a basic frame classifier and a regressor.

[0024] Further, the application provides a sample target detection method, wherein adjusting the trained classification network according to the basic category data and the new category data to obtain the calibrated classification network specifically comprises:

[0025] Constructing second support data and second query data according to the basic category data;

[0026] Obtaining a prototype of a second category according to the second support data;

[0027] Constructing a second category topology graph according to the prototype of the second category and obtaining second representative category feature knowledge according to the second category topology graph;

[0028] Determining a conversion prototype of a second foreground category, a conversion prototype of a second agent category and a corresponding second meta-loss according to the second representative category feature knowledge;

[0029] Obtaining a second region division of the second query data and a corresponding second refinement loss;

[0030] Determining a corresponding second initial ROI feature according to the second region division;

[0031] Determining a second enhanced ROI feature according to the second initial ROI feature and the second representative category feature knowledge;

[0032] Performing regression and classification on the second query data according to the second enhanced ROI feature to obtain a corresponding second regression loss and a second classification loss;

[0033] Determining a second calibration loss according to a trainable parameter of a basic frame classifier in the trained classification network, the conversion prototype of the second foreground category and the conversion prototype of the second agent category;

[0034] Determining a second overall loss according to the second meta-loss, the second refinement loss, the second regression loss, the second classification loss and the second calibration loss;

[0035] Adjusting the trainable parameters of the basic frame classifier and the regressor in the trained classification network to obtain a minimum second overall loss, and taking the trained classification network corresponding to the minimum second overall loss as a calibrated classification network.

[0036] Further, the application provides a sample target detection method, wherein the method comprises the following steps:

[0037] determining a first representative class feature knowledge conversion result when the first representative class feature knowledge is converted into the same dimension as the basic frame classifier parameters, and taking the first representative class feature knowledge conversion result as the conversion prototype of the first proxy class.

[0038] Further, the application provides a sample target detection method, wherein the method comprises the following steps:

[0039] determining a first conversion function according to which the first representative class feature knowledge is converted into the same dimension as the basic frame classifier parameters;

[0040] squeezing the first representative class feature knowledge to obtain a representative feature of the first proxy class;

[0041] determining a conversion prototype of the first proxy class according to the first conversion function and the representative feature of the first proxy class.

[0042] Further, the application provides a sample target detection method, wherein the method comprises the following steps:

[0043] determining a first local context knowledge according to the first initial ROI feature;

[0044] determining a first global context knowledge according to the first representative class feature knowledge;

[0045] determining a first enhanced ROI feature according to the first initial ROI feature, the first local context knowledge and the first global context knowledge.

[0046] In a second aspect, the application provides a sample target detection device, comprising:

[0047] a sample acquisition module, configured to acquire a sample to be detected;

[0048] a target class acquisition module, configured to input the sample to be detected into a calibrated classification network to obtain a corresponding target class.

[0049] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the sample target detection methods described above when executing the program.

[0050] The present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any of the sample target detection methods described above.

[0051] The present application provides a sample target detection method and device, which comprises the following steps of: obtaining a sample to be detected; and inputting the sample to be detected into a calibrated classification network to obtain a corresponding target category. The present application aggregates global and local context information in the process of training the calibrated classification network and calibrates the calibrated classification network based on category knowledge, so that the calibrated classification network can learn more robust boundaries for foreground and background categories, thereby determining the target category of the sample. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the present application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings described below are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0053] Figure 1 is a flowchart of the sample target detection method provided by the present application;

[0054] Figure 2 is a structural schematic diagram of the sample target detection device provided by the present application;

[0055] Figure 3 is a structural schematic diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0056] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0057] The present application provides a sample target detection method, comprising: Figure 1 The present application provides a sample target detection method, comprising:

[0058] Step 100: obtaining a sample to be detected;

[0059] Specifically, a sample image to be detected is acquired, which can be a large sample image with a large amount of target annotation information or a small sample image with only a few target annotation information.

[0060] Step 200: inputting the sample to be detected into the calibrated classification network to obtain a corresponding target category.

[0061] Specifically, the acquired sample to be detected is input into the calibrated classification network to obtain a corresponding target category output result.

[0062] The sample target detection method provided by the application comprises the following steps: acquiring a sample to be detected; and inputting the sample to be detected into a calibrated classification network to obtain a corresponding target category.

[0063] Further, the sample target detection method provided by the application comprises the following steps:

[0064] acquiring basic category data and new category data;

[0065] training an initial classification network according to the basic category data to obtain a trained classification network;

[0066] adjusting the trained classification network according to the basic category data and the new category data to obtain the calibrated classification network.

[0067] Specifically, the calibrated classification network is acquired by twice training in the embodiment of the application. The first training is based on the basic category data, wherein the basic category data C base refers to image data with multiple target annotation information for each category, and the basic bounding box classifier network is trained to obtain category-independent parameters, i.e. the parameters of the CNN main body, RPN, category graph network, heterogeneous graph network, foreground and background generation network.

[0068] The second training is based on the basic category data and the new category data C novelwhere the new class data is image data with only a few target annotation information for each class, which is used to update only the parameters of the box predictor (i.e., the box regressor and the classifier). Specifically, when a new class is added in the fine-tuning stage, i.e., in the second training process, the foreground class representation is first updated using the fixed parameters of the graph network and the background class representation is generated. Next, the prototype-based box classifier is predicted using the mapping function f, which is then used to calibrate the base box classifier. This process can be seen as a special case of knowledge distillation: the prototype classifier parameters predicted by the fixed parameters of the graph network and the mapping function f learned in the pre-training stage ('teacher') become the distillation target for the RoI-based box classifier learning in the fine-tuning stage ('student').

[0069] In the pre-training, i.e., the first training stage, by aligning the prototype-predicted box classifier parameters with the base box classifier parameters learned under the guidance of ground truth, this is similar to the idea of the zero-shot learning method that uses class semantic information to predict the parameters of new classes without any annotated samples. Since there are enough labeled samples in the pre-training stage, the initial classification network can learn the class-independent parameters related to the generation of the prototype box classifier parameters well, so that the calibrated classification network after training can be calibrated in the fine-tuning stage to improve the object detection performance.

[0070] Further, according to the sample object detection method provided by the present application, wherein the initial classification network is trained according to the basic class data to obtain a trained classification network, specifically comprising:

[0071] According to the basic class data, a first support data and a first query data are constructed;

[0072] According to the first support data, a prototype of the first class is obtained;

[0073] According to the first class topology graph, a first representative class feature knowledge is obtained;

[0074] According to the first representative class feature knowledge, a first foreground class conversion prototype, a first proxy class conversion prototype and a corresponding first meta-loss are determined;

[0075] The first query data is obtained, and a first region division and a corresponding first refinement loss are obtained;

[0076] According to the first region division, a corresponding first initial ROI feature is determined;

[0077] According to the first initial ROI feature and the first representative class feature knowledge, a first enhanced ROI feature is determined;

[0078] Regressing and classifying the first query data according to the first enhanced ROI feature to obtain a corresponding first regression loss and a first classification loss;

[0079] determining a first calibration loss based on trainable parameters of a basic box classifier in the initial classification network, a conversion prototype of the first foreground class, and a conversion prototype of the first proxy class;

[0080] determining a first overall loss based on the first meta-loss, the first refinement loss, the first regression loss, the first classification loss, and the first calibration loss;

[0081] Adjust the parameters of the initial classification network to obtain the minimum first overall loss, use the initial classification network corresponding to the minimum first overall loss as the trained classification network, and fix the fully connected layer parameters of the trained classification network.

[0082] Specifically, during the training process, c ,c∈C base} Extract K for each category p (K p =200) to build supporting data Used to learn the initial prototype representation of the category. If the image selected in the training step contains an object of the new category, the corresponding area is considered as the background category. Since there are multiple objects per image, the number of images selected in these steps may be less than K p During the training phase, the detector is optimized with the help of category prototypes learned from support data to correctly localize and classify objects in query images.

[0083] Faster R-CNN is a popular two-stage object detection framework that has achieved great success in general object detection tasks over the past few years. In the first stage, the RPN uses the feature maps generated by the convolutional backbone network (ResNet-101) to generate candidate regions of interest (ROIs). In the second stage, RoI alignment is used to extract RoI (Region of Interest) features from these candidate regions. Finally, the RoI features are used to locate and classify objects in the image. In this paper, this two-stage framework is used to extract RoI features from the query image.

[0084] During this training process, we construct the first support data and extract the prototype of the category, thereby introducing discriminative information of the category to help locate and classify the target in the query image. For the first support data, we first use the convolutional backbone network (ResNet-101) to extract the visual features of the target from the feature map obtained by cropping the bounding box. Then, we average the target features of different categories to obtain the corresponding first category prototype. Where d is the feature dimension and N is the number of categories.

[0085] A class topology graph is constructed using the initial class prototypes generated in the above process to propagate class information to learn first representative class knowledge Specifically, first, the similarity between the first class prototypes is calculated using an attention mechanism to generate an adjacency matrix A:

[0086] A p = softmax(PW0(PW0) T ) (1)

[0087] where T denotes the transpose operation, and PW0is a trainable weight matrix. Then, a message passing operation is performed using A p to learn second representative class knowledge:

[0088]

[0089] where, and PW0is a trainable weight matrix, and σ denotes a nonlinear activation function. The resulting first representative class feature knowledge will be used in the subsequent context aggregation and classifier calibration process.

[0090] To facilitate the distillation of class knowledge, the first representative class feature knowledge is first converted into first foreground class converted prototypes with the same dimension as the parameters of the base box classifier

[0091]

[0092] where is a transformation function consisting of a fully connected layer and a ReLU activation. To ensure that the converted first foreground class converted prototypes are discriminative, the initial classification network correctly classifies them by a prototype classifier.

[0093] It is worth noting that the representative class prototypes are only learned for the first foreground classes. Therefore, the class knowledge can only be used to calibrate the base box classifier for the foreground classes. However, unlike general few-shot learning, FSOD needs to robustly recognize the first background class. Therefore, by defining a proxy class to represent the main features of all first foreground classes, and making the classifier parameters of the first background class far away from the converted prototypes of the proxy class. Specifically, by first squeezing the first representative class knowledge to learn the representative features G p of the proxy class at each training step:

[0094]

[0095] wherein and are weight matrices, is an assignment matrix. Then, the first query image is transformed into a first proxy class conversion prototype by using the same transformation function f. Next, the basic box classifier parameters of the first background class are calibrated to be orthogonal to the first proxy class conversion prototype C b . By this scheme, the difference between the first background class and the first foreground class can be enlarged. And the cross-entropy loss L meta is used to encourage each prototype to belong to its corresponding class when determining the conversion prototype of the first foreground class and the conversion prototype of the first proxy class according to the first representative class feature knowledge.

[0096] The first query image is input into the initial classification network RPN for outputting a refined first region division and a corresponding first refined loss L rpn .

[0097] According to the first region division, a RoI alignment operation is used to extract first initial RoI (Region of Interest) features from the candidate regions. wherein n represents the number of RoIs extracted in the picture. In order to capture rich context information to enhance the RoI features, a heterogeneous graph is constructed to model the region-region and region-class relationship information. Specifically, for each RoI, the region-region relationship is used to aggregate the information of the remaining regions in the query image to obtain local context knowledge At the same time, the region-class relationship information is propagated to capture global context knowledge C c .

[0098]

[0099]

[0100] wherein and represent the region-region and region-class adjacency matrices. W2, W3, W4 and W5 are trainable weight matrices. By concatenating the global and local context information to enhance the RoI features:

[0101]

[0102] wherein [*,*,*] represents a connection operation, which is performed in a row-splicing manner. is the enhanced RoI feature, i.e., the first enhanced ROI feature.

[0103] According to the first enhanced ROI feature, the first query data is regressed and classified, and a corresponding first regression loss, a classification result of the first query data and a first classification loss are obtained. The first classification loss L cls is the loss of the classifier of the cross-entropy basic frame classifier, the first regression loss L reg is the smooth L1 loss of the regressor of the basic frame classifier.

[0104] The labeled objects of the new class are few and cannot well represent the class distribution. Therefore, it is not enough to rely on the RoI feature to train the frame classifier. In the present application, the parameters of the classifier are calibrated by using class knowledge. Based on the RoI feature obtained in the above steps, the basic frame classifier can be learned, which is composed of a fully connected layer and a Softmax activation. The trainable parameters of the basic frame classifier (for N foreground classes and 1 background class) are represented as The calibration loss L cal is defined as follows:

[0105]

[0106] where ||*|| represents the L2 norm, and · represents the dot product. p is the trainable weight matrix of the basic frame classifier in the initial classification network. p [N+1] represents the parameters of the first background class, while p [i] represents the parameters of the first foreground class i. p and C b are the conversion prototypes of the first foreground class and the conversion prototypes of the first proxy class.

[0107] The proposed initial classification network is trained by minimizing the following loss function:

[0108] L = aL det + bL meta + gL cal (9)

[0109] where a, b and g are hyperparameters, which are set to 1.0, 1.0 and 0.5 respectively in the experiment. det , L meta and L cal are the general detection loss, the meta loss and the calibration loss respectively.

[0110] Detection loss. The following detection loss function is used according to the existing method:

[0111] L det = L cls + L reg + L rpn (10)

[0112] After obtaining the minimum value of the loss function, the corresponding initial classification network is taken as a trained classification network, and other training parameters in the trained classification network except the basic frame classifier and the regressor are fixed. That is, the weight parameters and full connection layer parameters in the above formulas are fixed.

[0113] Further, the application provides a sample target detection method, wherein the trained classification network is adjusted according to the basic category data and the new category data to obtain the calibrated classification network, and the method specifically comprises the following steps.

[0114] Second support data and second query data are constructed according to the basic category data;

[0115] A prototype of a second category is obtained according to the second support data;

[0116] A second category topology graph is constructed according to the prototype of the second category, and second representative category feature knowledge is obtained according to the second category topology graph;

[0117] A conversion prototype of a second foreground category, a conversion prototype of a second agent category and a corresponding second meta-loss are determined according to the second representative category feature knowledge;

[0118] Second region division of the second query data and a corresponding second refinement loss are obtained;

[0119] Second initial ROI features are determined according to the second region division;

[0120] Second enhanced ROI features are determined according to the second initial ROI features and the second representative category feature knowledge;

[0121] The second query data are regressed and classified according to the second enhanced ROI features to obtain a corresponding second regression loss and a second classification loss;

[0122] A second calibration loss is determined according to the trainable parameters of the basic frame classifier in the trained classification network, the conversion prototype of the second foreground category and the conversion prototype of the second agent category;

[0123] A second overall loss is determined according to the second meta-loss, the second refinement loss, the second regression loss, the second classification loss and the second calibration loss;

[0124] Trainable parameters of the basic frame classifier and the regressor in the trained classification network are adjusted to obtain a minimum second overall loss, and the trained classification network corresponding to the minimum second overall loss is taken as a calibrated classification network.

[0125] Specifically, one is composed of the basic category C base Each category has a large amount of target annotation information The other is composed of the new category C novel Each category has only a few target annotations (i.e., ). Where y={(cls l ,box l )|l∈obj x} represents each target l∈obj in image x x The class label cls l And bounding box annotation information box l In order to facilitate the calculation of the prototype of each category c, the image with category c target is used to construct the category data set D corresponding to the category c ={(x,box l )|cls l =c,l∈obj x ,x∈D base ∪D novel}.

[0126] During the training process, c ,c∈C}where(C=C base ∪C novel Extract K for each category f targets (i.e., K f shot settings) to build support data Used to learn the initial prototype representation of the category. Since there are multiple objects per image, the number of images selected in these steps may be less than K f During the training phase, the detector is optimized with the help of category prototypes learned from support data to correctly localize and classify objects in query images.

[0127] In the training process of the embodiment of the present invention, the training is performed based on the trained classification network. During the training process, the parameters in the fully connected layer are fixed in the trained classification network and therefore are not changed in this step of the training process.

[0128] Faster R-CNN is a popular two-stage object detection framework that has achieved great success in general object detection tasks over the past few years. In the second stage, the RPN uses the feature maps generated by the convolutional backbone network (ResNet-101) to generate candidate regions of interest for the object. The second stage uses the RoI alignment operation to extract RoI (Region of Interest) features from these candidate regions. Finally, the RoI features are used to locate and classify objects in the image. In this paper, this two-stage framework is used to extract RoI features for the query image.

[0129] In the training process, the second support data is constructed, and the prototypes of the categories are extracted to introduce the discriminative information of the categories to help the positioning and classification of the target in the query image. For the second support data, the visual features of the target are first extracted from the feature map cropped according to the bounding box using the convolutional backbone network (ResNet-101). Then, the target features of different categories are averaged respectively to obtain the corresponding second category prototypes where d is the feature dimension, and N is the number of categories.

[0130] The initial category prototypes generated in the above process are used to construct a category topology graph to propagate category information to learn the second representative category knowledge Specifically, first, the similarity between the second category prototypes is calculated using the attention mechanism to generate an adjacency matrix A:

[0131] A p = softmax(P'W0(P'W0) T ) (11)

[0132] where T represents the transpose operation, is a trainable weight matrix. Then, the message passing operation is performed using A p to learn the second representative category knowledge:

[0133]

[0134] where, is a trainable weight matrix, and σ represents a nonlinear activation function. The obtained second representative category feature knowledge will be used in the subsequent context aggregation and classifier calibration process.

[0135] In order to facilitate the extraction of category knowledge, the second representative category feature knowledge is first converted into the second foreground category conversion prototype with the same dimension as the parameters of the basic frame classifier

[0136]

[0137] where is a transformation function consisting of a fully connected layer and a ReLU activation. In order to ensure that the converted second foreground category conversion prototype has discriminative, the initial classification network correctly classifies them through the prototype classifier.

[0138] It is worth noting that the representative category prototype Only the second foreground categories are learned. Thus, the category knowledge can only be used to calibrate the base box classifier of the foreground categories. However, unlike general few-shot learning, FSOD requires robust recognition of the second background category. Therefore, by defining a proxy category to represent the main features of all second foreground categories, and making the classifier parameters of the second background category far away from the proxy category's transformation prototype. Specifically, by first squeezing the second representative category knowledge to learn the representative features C p

[0139]

[0140] where and are weight matrices, is an assignment matrix. Then, transform to the second proxy category transformation prototype C Next, by calibrating the classifier parameters of the second background category to be orthogonal to the second proxy category transformation prototype C b , the difference between the second background category and the second foreground categories can be enlarged. And by using cross-entropy loss L meta when determining the transformation prototype of the second foreground category according to the second representative category feature knowledge, and the transformation prototype of the second proxy category, each prototype is encouraged to belong to its corresponding category.

[0141] Input the second query image into the initial classification network RPN to output the second region division refined, and the corresponding second refined loss L rpn .

[0142] According to the second region division, the second initial RoI (Region of Interest) features are extracted from these candidate regions using the RoI alignment operation. where n represents the number of RoIs extracted in the picture. In order to capture rich context information to enhance the RoI features, a heterogeneous graph is constructed to model the region-region and region-category relationship information. Specifically, for each RoI, the region-region relationship is used to aggregate the information of the remaining regions in the query image to obtain local context knowledge At the same time, the region-category relationship information is propagated to capture global context knowledge C c .

[0143]

[0144]

[0145] where and​ Region-Region and Region-Category adjacency matrix. W2, W3, W4 and W5 are trainable weight matrices. By concatenating global and local context information to augment RoI features:

[0146]

[0147] where [*,*,*] denotes concatenation operation, performed in a row-wise manner. is the augmented RoI feature, i.e., the second augmented ROI feature.

[0148] According to the second augmented ROI feature, the second query data is regressed and classified to obtain a corresponding second regression loss, a classification result of the second query data and a second classification loss. The second classification loss L cls ′ is the loss of the classifier of the basic box classifier, the second regression loss L reg ′ is the smooth L1 loss of the regressor of the basic box classifier.

[0149] The labeled objects of the new category are few and cannot well represent the category distribution. Therefore, it is not enough to rely on RoI features to train the box classifier. In the present application, the parameters of the classifier are calibrated using category knowledge. Based on the RoI features obtained in the above steps, the basic box classifier can be learned, which consists of a fully connected layer and a Softmax activation. The trainable parameters of the basic box classifier (for N foreground categories and 1 background category) are denoted as The calibration loss L cal ′ is defined as follows:

[0150]

[0151] where ||*|| denotes L2 norm, and · denotes dot product. I p ′ is the trainable weight matrix of the basic box classifier in the initial classification network. I p ′ [N+1] represents the parameters of the second background category, while I p ′ [i] represents the parameters of the second foreground category i. C p ′ and C b ′ are the conversion prototypes of the second foreground categories and the conversion prototypes of the second proxy categories.

[0152] The proposed trained classification network is trained by minimizing the following loss function:

[0153] L′=αL det ′+βL meta ′+γL cal ′ (19)

[0154] where a, b and g are hyperparameters which are set to 1.0, 1.0 and 0.5 respectively in the experiment. L det ', L meta ', and L cal are the general detection loss, meta-loss and calibration loss respectively.

[0155] Detection loss. The following detection loss function is adopted according to the existing method:

[0156] L det ' = L cls + L reg + L rpn '(20)

[0157] After obtaining the minimum value of the loss function, the corresponding trained classification model is taken as a trained classification network, and the full connection layer parameters of the trained classification network are fixed.

[0158] Further, the application provides a sample target detection method, wherein the first representative class feature knowledge is used to determine a first proxy class conversion prototype, and the method specifically comprises the following steps.

[0159] The first representative class feature knowledge is converted into a first representative class feature knowledge conversion result with the same dimension as the basic frame classifier parameter, and the first representative class feature knowledge conversion result is taken as the first proxy class conversion prototype.

[0160] Specifically, in order to facilitate the extraction of class knowledge, first, the first representative class feature knowledge is converted into a first proxy class conversion prototype with the same dimension as the basic frame classifier parameter.

[0161]

[0162] wherein is a transformation function composed of a full connection layer and a ReLU activation. In order to ensure that the converted first proxy class conversion prototype has discriminability, the initial classification network correctly classifies them through the prototype classifier.

[0163] Further, the application provides a sample target detection method, wherein the first representative class feature knowledge is used to determine a first proxy class conversion prototype, and the method specifically comprises the following steps.

[0164] The first conversion function used to convert the first representative class feature knowledge into a first proxy class conversion prototype with the same dimension as the basic frame classifier parameter is determined.

[0165] ​extruding the first representative class feature knowledge to obtain a representative feature of a first proxy class;

[0166] determining a conversion prototype of the first proxy class according to the first conversion function and the representative feature of the first proxy class.

[0167] Specifically, it is worth noting that the representative class prototype Only the first foreground class is learned. Therefore, the class knowledge can only be used to calibrate the basic box classifier of the foreground class. However, unlike general few-shot learning, FSOD needs to robustly identify the first background class. Therefore, by defining a proxy class to represent the main features of all first foreground classes, and making the classifier parameters of the first background class far away from the conversion prototype of the proxy class. Specifically, by first extruding the first representative class feature knowledge to learn the representative feature G of the proxy class p :

[0168]

[0169] wherein and is a weight matrix, is an assignment matrix. Then, use the same transformation function f to transform into the first proxy class conversion prototype Next, by calibrating the classifier parameters of the first background class, it is orthogonal to the first proxy class conversion prototype C b . Through this scheme, the difference between the first background class and the first foreground class can be expanded.

[0170] Further, according to the sample target detection method provided by the present application, wherein the first enhanced ROI feature is determined according to the first initial ROI feature and the first representative class feature knowledge, specifically comprising:

[0171] determining a first local context knowledge according to the first initial ROI feature;

[0172] determining a first global context knowledge according to the first representative class feature knowledge;

[0173] determining a first enhanced ROI feature according to the first initial ROI feature, the first local context knowledge and the first global context knowledge.

[0174] Specifically, according to the first region division, the RoI alignment operation is used to extract the first initial RoI (Region of Interest) feature from these candidate regions. where n denotes the number of extracted RoIs in the image. To capture rich context information to enhance RoI features, a heterogeneous graph is constructed to model the region-region and region-class relationship information. Specifically, for each RoI, the region-region relationship is utilized to aggregate the information of the remaining regions in the query image to obtain the local context knowledge R i c Meanwhile, the region-class relationship information is propagated to capture the global context knowledge C c .

[0175]

[0176]

[0177] where and denote the adjacency matrices of region-region and region-class. W2, W3, W4 and W5 are trainable weight matrices. By concatenating the global and local context information to enhance RoI features:

[0178]

[0179] where [*,*,*] denotes the concatenation operation, which is performed in a row-wise concatenation manner. is the enhanced RoI feature, i.e., the first enhanced ROI feature.

[0180] In combination with Figure 2 , the present application provides a sample target detection device, comprising:

[0181] a sample acquisition module 21 configured to acquire a sample to be detected;

[0182] a target class acquisition module 22 configured to input the sample to be detected into a calibrated classification network to obtain a corresponding target class.

[0183] Since the device provided by the embodiments of the present application can be used to perform the method described in the above embodiments, the working principle and beneficial effects are similar, and therefore will not be described in detail here. For specific content, please refer to the introduction of the above embodiments.

[0184] The sample target detection device provided by the present application acquires a sample to be detected, and inputs the sample to be detected into a calibrated classification network to obtain a corresponding target class. In the present application, the global and local context information is aggregated in the process of training the calibrated classification network, and the calibrated classification network is calibrated based on the class knowledge. The calibrated classification network can learn more robust boundaries for foreground and background classes, thereby determining the target class of the sample.

[0185] Further, the application provides a sample target detection device, wherein the target category acquisition module 22 is configured to:

[0186] acquire basic category data and new category data;

[0187] train an initial classification network according to the basic category data to obtain a trained classification network;

[0188] adjust the trained classification network according to the basic category data and the new category data to obtain the calibrated classification network.

[0189] Further, the application provides a sample target detection device, wherein the target category acquisition module 22 is configured to:

[0190] construct first support data and first query data according to the basic category data;

[0191] acquire a prototype of a first category according to the first support data;

[0192] construct a first category topology graph according to the prototype of the first category, and acquire first representative category feature knowledge according to the first category topology graph;

[0193] determine a conversion prototype of a first foreground category, a conversion prototype of a first proxy category and a corresponding first meta-loss according to the first representative category feature knowledge;

[0194] acquire a first region division of the first query data and a corresponding first refinement loss;

[0195] determine a corresponding first initial ROI feature according to the first region division;

[0196] determine a first enhanced ROI feature according to the first initial ROI feature and the first representative category feature knowledge;

[0197] regress and classify the first query data according to the first enhanced ROI feature to obtain a corresponding first regression loss and a first classification loss;

[0198] determine a first calibration loss according to trainable parameters of a basic frame classifier in the initial classification network, the conversion prototype of the first foreground category and the conversion prototype of the first proxy category;

[0199] determine a first overall loss according to the first meta-loss, the first refinement loss, the first regression loss, the first classification loss and the first calibration loss;

[0200] Adjusting parameters of the initial classification network to obtain a minimum first overall loss, taking the initial classification network corresponding to the minimum first overall loss as a trained classification network and fixing full connection layer parameters of the trained classification network.

[0201] Further, the application provides a sample target detection device, wherein the target category acquisition module 22 is configured to:

[0202] Second support data and second query data are constructed according to the basic category data;

[0203] A prototype of a second category is acquired according to the second support data;

[0204] A second category topology graph is constructed according to the prototype of the second category, and second representative category feature knowledge is acquired according to the second category topology graph;

[0205] A conversion prototype of a second foreground category, a conversion prototype of a second agent category and a corresponding second meta-loss are determined according to the second representative category feature knowledge;

[0206] Second region division of the second query data and a corresponding second refinement loss are acquired;

[0207] A corresponding second initial ROI feature is determined according to the second region division;

[0208] A second enhanced ROI feature is determined according to the second initial ROI feature and the second representative category feature knowledge;

[0209] The second query data is regressed and classified according to the second enhanced ROI feature, to obtain a corresponding second regression loss and a second classification loss;

[0210] A second calibration loss is determined according to trainable parameters of a basic frame classifier in the trained classification network, the conversion prototype of the second foreground category and the conversion prototype of the second agent category;

[0211] A second overall loss is determined according to the second meta-loss, the second refinement loss, the second regression loss, the second classification loss and the second calibration loss;

[0212] Trainable parameters of the basic frame classifier and the regressor in the trained classification network are adjusted to obtain a minimum second overall loss, and the trained classification network corresponding to the minimum second overall loss is taken as a calibrated classification network.

[0213] Further, the application provides a sample target detection device, wherein the target category acquisition module 22 is configured to:

[0214] Determine a result of converting the first representative category feature knowledge into the first representative category feature knowledge when the first representative category feature knowledge has the same dimension as the basic box classifier parameter, and use the result of converting the first representative category feature knowledge as a conversion prototype of the first foreground category.

[0215] Furthermore, according to a sample target detection method provided by the present invention, the step of determining a conversion prototype of a first proxy category based on the first representative category feature knowledge specifically includes:

[0216] Determining a first conversion function based on which the first representative category feature knowledge is converted into a parameter having the same dimension as the basic box classifier;

[0217] Squeezing the first representative category feature knowledge to obtain representative features of the first agent category;

[0218] A conversion prototype of the first agent category is determined according to the first conversion function and representative characteristics of the first agent category.

[0219] Furthermore, according to a sample target detection device provided by the present invention, the target category acquisition module 22 is configured to:

[0220] determining first local context knowledge according to the first initial ROI feature;

[0221] determining first global context knowledge based on the first represented category feature knowledge;

[0222] A first enhanced ROI feature is determined according to the first initial ROI feature, the first local context knowledge, and the first global context knowledge.

[0223] Figure 3 An example of a physical structure diagram of an electronic device is shown below. Figure 3 As shown, the electronic device may include: a processor 310, a communications interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communications interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 may call logic instructions in the memory 330 to execute a sample target detection method, which includes: obtaining a sample to be detected; inputting the sample to be detected into a calibrated classification network to obtain a corresponding target category.

[0224] In addition, the logic instructions in the memory 330 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0225] In another aspect, the present application also provides a computer program product, which comprises a computer program stored on a non-transitory computer readable storage medium, and the computer program comprises program instructions, and when the program instructions are executed by a computer, the computer can execute a sample target detection method provided by the above-mentioned method, and the method comprises: obtaining a sample to be detected; inputting the sample to be detected into a calibrated classification network to obtain a corresponding target category.

[0226] In another aspect, the present application also provides a computer program product, which comprises a computer program stored on a non-transitory computer readable storage medium, and the computer program comprises program instructions, and when the program instructions are executed by a computer, the computer can execute a sample target detection method provided by the above-mentioned method, and the method comprises: obtaining a sample to be detected; inputting the sample to be detected into a calibrated classification network to obtain a corresponding target category.

[0227] The device embodiments described above are only schematic, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment. Those skilled in the art can understand and implement it without creative labor.

[0228] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0229] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A sample target detection method, characterized in that: include: Obtaining a sample image to be detected; Inputting the sample to be detected into the calibrated classification network to obtain the corresponding target category; The calibrated classification network is trained by the following method: Get basic category data and new category data; Training the initial classification network according to the basic category data to obtain a trained classification network; Adjusting the trained classification network according to the basic category data and the new category data to obtain the calibrated classification network; The training of the initial classification network according to the basic category data to obtain a trained classification network specifically includes: constructing first supporting data and first query data according to the basic category data; Acquire a prototype of a first category according to the first supporting data; constructing a first category topology map based on the prototype of the first category, and acquiring first representative category feature knowledge based on the first category topology map; Determining a conversion prototype of a first foreground category, a conversion prototype of a first proxy category, and a corresponding first element loss based on the first representative category feature knowledge; Obtaining a first region division and a corresponding first refinement loss of the first query data; Determine a corresponding first initial ROI feature according to the first region division; determining a first enhanced ROI feature according to the first initial ROI feature and the first represented category feature knowledge; Regressing and classifying the first query data according to the first enhanced ROI feature to obtain a corresponding first regression loss and first classification loss; determining a first calibration loss based on trainable parameters of a basic box classifier in the initial classification network, a conversion prototype of the first foreground class, and a conversion prototype of the first proxy class; determining a first overall loss based on the first meta-loss, the first refinement loss, the first regression loss, the first classification loss, and the first calibration loss; Adjusting the parameters of the initial classification network to obtain a minimum first overall loss, using the initial classification network corresponding to the minimum first overall loss as the trained classification network and fixing the training parameters of the trained classification network except for the basic box classifier and the regressor; The step of determining the conversion prototype of the first foreground category based on the first representative category feature knowledge specifically includes: Determining a result of converting the first representative category feature knowledge into the first representative category feature knowledge when the first representative category feature knowledge has the same dimension as the basic box classifier parameter, and using the result of converting the first representative category feature knowledge as a conversion prototype of the first foreground category; The determination of converting the first representative category feature knowledge into a result after the first representative category feature knowledge is converted into the first representative category feature knowledge having the same dimension as the basic box classifier parameter is expressed as follows: ； in, , represents the conversion prototype of the first foreground category; , represents the transformation function consisting of a fully connected layer and ReLU activation; Represents the first representative category feature knowledge; The step of determining the conversion prototype of the first agent category based on the first representative category feature knowledge specifically includes: Determining a first conversion function based on which the first representative category feature knowledge is converted into a value having the same dimension as the basic box classifier parameter; Squeezing the first representative category feature knowledge to obtain representative features of the first agent category; A conversion prototype of the first agent category is determined according to the first conversion function and representative characteristics of the first agent category.

2. The sample target detection method according to claim 1, characterized in that: The step of adjusting the trained classification network according to the basic category data and the new category data to obtain the calibrated classification network specifically includes: constructing second supporting data and second query data according to the basic category data; obtaining a prototype of a second category according to the second supporting data; constructing a second category topology map based on the prototype of the second category, and acquiring second representative category feature knowledge based on the second category topology map; Determining a conversion prototype of a second foreground category, a conversion prototype of a second proxy category, and a corresponding second element loss based on the second representative category feature knowledge; Obtaining a second region division and a corresponding second refinement loss for the second query data; Determine a corresponding second initial ROI feature according to the second region division; determining a second enhanced ROI feature based on the second initial ROI feature and the second representative category feature knowledge; Regressing and classifying the second query data according to the second enhanced ROI feature to obtain a corresponding second regression loss and second classification loss; determining a second calibration loss based on trainable parameters of a basic box classifier in the trained classification network, a conversion prototype of the second foreground class, and a conversion prototype of the second proxy class; determining a second overall loss based on the second element loss, the second refinement loss, the second regression loss, the second classification loss, and the second calibration loss; Adjusting the trainable parameters of the basic box classifier and the regressor in the trained classification network to obtain a minimum second overall loss, and using the trained classification network corresponding to the minimum second overall loss as the calibrated classification network; The determining of the conversion prototype of the second foreground category, the conversion prototype of the second proxy category, and the corresponding second element loss based on the second representative category feature knowledge specifically includes: The conversion prototype of the second foreground category is expressed as follows: ； in, a transformed prototype representing the second foreground category; represents a transformation function consisting of a fully connected layer and ReLU activation; represents the second representational category knowledge; The determining of the conversion prototype of the second foreground category, the conversion prototype of the second proxy category, and the corresponding second element loss based on the second representative category feature knowledge specifically includes: determining a second conversion function based on which the second representative category feature knowledge is converted into a value having the same dimension as the basic box classifier parameter; Squeezing the second representative category feature knowledge to obtain representative features of the second agent category; A conversion prototype of the second agent category is determined according to the second conversion function and representative characteristics of the second agent category.

3. The sample target detection method according to claim 1, characterized in that: The determining of a first enhanced ROI feature according to the first initial ROI feature and the first representative category feature knowledge specifically includes: determining first local context knowledge according to the first initial ROI feature; determining first global context knowledge based on the first represented category feature knowledge; A first enhanced ROI feature is determined according to the first initial ROI feature, the first local context knowledge, and the first global context knowledge.

4. A sample target detection device, characterized in that: include: A sample acquisition module is used to acquire the sample image to be detected; A target category acquisition module is used to input the sample to be detected into a calibrated classification network to obtain the corresponding target category; The target category acquisition module is specifically used to: Get basic category data and new category data; Training the initial classification network according to the basic category data to obtain a trained classification network; Adjusting the trained classification network according to the basic category data and the new category data to obtain the calibrated classification network; The target category acquisition module is specifically used to: constructing first supporting data and first query data according to the basic category data; Acquire a prototype of a first category according to the first supporting data; constructing a first category topology map based on the prototype of the first category, and acquiring first representative category feature knowledge based on the first category topology map; Determining a conversion prototype of a first foreground category, a conversion prototype of a first proxy category, and a corresponding first element loss based on the first representative category feature knowledge; Obtaining a first region division and a corresponding first refinement loss of the first query data; Determine a corresponding first initial ROI feature according to the first region division; determining a first enhanced ROI feature according to the first initial ROI feature and the first represented category feature knowledge; Regressing and classifying the first query data according to the first enhanced ROI feature to obtain a corresponding first regression loss and first classification loss; determining a first calibration loss based on trainable parameters of a basic box classifier in the initial classification network, a conversion prototype of the first foreground class, and a conversion prototype of the first proxy class; determining a first overall loss based on the first meta-loss, the first refinement loss, the first regression loss, the first classification loss, and the first calibration loss; Adjusting the parameters of the initial classification network to obtain a minimum first overall loss, using the initial classification network corresponding to the minimum first overall loss as the trained classification network and fixing the training parameters of the trained classification network except for the basic box classifier and the regressor; The target category acquisition module is specifically used to: Determining a result of converting the first representative category feature knowledge into the first representative category feature knowledge when the first representative category feature knowledge has the same dimension as the basic box classifier parameter, and using the result of converting the first representative category feature knowledge as a conversion prototype of the first foreground category; The determination of converting the first representative category feature knowledge into a result after the first representative category feature knowledge is converted into the first representative category feature knowledge having the same dimension as the basic box classifier parameter is expressed as follows: ； in, , represents the conversion prototype of the first foreground category; , represents the transformation function consisting of a fully connected layer and ReLU activation; Represents the first representative category feature knowledge; The target category acquisition module is specifically used to: Determining a first conversion function based on which the first representative category feature knowledge is converted into a value having the same dimension as the basic box classifier parameter; Squeezing the first representative category feature knowledge to obtain representative features of the first agent category; A conversion prototype of the first agent category is determined according to the first conversion function and representative characteristics of the first agent category.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the sample target detection method according to any one of claims 1 to 3 are implemented.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the sample target detection method according to any one of claims 1 to 3 are implemented.

Images