Target detection model training method and device

Abstract

The invention discloses a target detection model training method and device. The method comprises the steps of obtaining instance level features of a target contained in a training image based on a detector; performing full connection layer classification processing based on the instance level features to obtain a first classification result; performing prototype classification processing based on the instance level features and the first classification result to obtain a second classification result; and training the detector based on the first classification result and the second classification result to obtain a target detection model. According to the scheme, on the basis of full connection layer classification, prototype classification processing is added, and the two classification modes jointly act on model training; wherein the prototype classification enables the feature alignment processing in the training process and the classification processing based on the pseudo tag to realize more sufficient interaction, and a target detection model with good domain adaptability is obtained.

Description

Technical Field

[0001] The present application relates to the technical field of target detection in autonomous driving perception algorithms, and more specifically, to a training method and device for a target detection model. Background Art

[0002] At present, deep learning-based object detection methods can achieve good performance in supervised settings. However, in actual applications, the application scenarios of the deployed models and the data used for training often have domain shifts due to different distributions. Figure 1 As shown in the figure, a model trained on clear-sky data significantly degrades when tested directly on foggy images. The most effective solution to alleviate this problem is to collect data from the target scene and manually annotate it, but this incurs significant time and labor costs. Domain-adaptive object detection, on the other hand, allows access to target domain data (unlabeled data) during the training phase, transferring source domain knowledge to the target domain, improving the robustness of the model. Currently, there are three mainstream solutions for domain-adaptive object detection: feature alignment based on adversarial learning, self-training methods based on pseudo-labeling, and a fusion of the two.

[0003] However, under the premise of the influence of weather, time period, etc. in autonomous driving scenarios, the above-mentioned various implementation methods often show that the distribution variance of features of the same type is large at the feature level. In addition, although the feature alignment and pseudo-label fusion methods represented by Adaptive Teacher AT (Adaptive Teacher, abbreviated as AT, a relatively classic domain adaptive target detection algorithm, which adopts a teacher-student model and uses image-level alignment to align the features of the target domain and the source domain during training, and uses the pseudo-labels given by the teacher model as the true value labels to train the student model) have achieved the adaptation of the detection model to the target domain to a certain extent, the fusion methods of feature alignment and pseudo-labels such as AT are relatively simple, and it is difficult to achieve ideal results. Summary of the Invention

[0004] In view of this, this application provides the following technical solutions:

[0005] The first aspect of the present application provides a method for training an object detection model, comprising:

[0006] Obtain instance-level features of the objects contained in the training image based on the detector;

[0007] Performing a fully connected layer classification process based on the instance-level features to obtain a first classification result;

[0008] Performing prototype classification processing based on the instance-level features and the first classification result to obtain a second classification result;

[0009] The detector is trained based on the first classification result and the second classification result to obtain a target detection model.

[0010] In one possible implementation, the first classification result includes a category and a classification score, and performing prototype classification based on the instance-level features and the first classification result to obtain a second classification result includes:

[0011] Obtaining a classification result in which the classification score of a category in the first classification result reaches a set value, and using the classification result as a pseudo label set;

[0012] Calculate the average of the instance-level features corresponding to all pseudo labels under each category in the classification result, and use the obtained average as the prototype of the current category;

[0013] Classify the instance-level features of the target contained in the training image based on the prototype to obtain a second classification result.

[0014] In one possible implementation, classifying the instance-level features of the object contained in the training image based on the prototype to obtain a second classification result includes:

[0015] The instance-level features of the objects contained in the training image are constructed based on the cosine distance to perform prototype-based classification, thereby obtaining a second classification result.

[0016] In one possible implementation, the following is further included:

[0017] During the training of the detector, the prototype of the corresponding category is updated according to the newly appeared instance-level features after each iteration.

[0018] In one possible implementation, the training of the detector based on the first classification result and the second classification result to obtain the target detection model includes:

[0019] Determining an overall loss of the detector, the overall loss comprising at least a fully connected classification cross entropy loss and a prototype classification cross entropy loss;

[0020] The detector is adjusted based on the overall loss to obtain a target detection model.

[0021] In one possible implementation, determining the overall loss of the detector includes:

[0022] Determine the fully connected categorical cross entropy loss based on the first classification result of the target instance-level features;

[0023] Determine the cross entropy loss based on the prototype classification based on the second classification result of the target instance-level features and the corresponding pseudo-label;

[0024] Performing consistency constraints on the first classification result and the second classification result to obtain a constraint loss;

[0025] The overall loss of the detector is determined based on the fully connected classification cross entropy loss, the prototype classification based cross entropy loss, the constraint loss and other detection model related losses.

[0026] In one possible implementation, the target detection model is a student model. After obtaining the target detection model, the method further includes:

[0027] The parameters of the student model are updated to the teacher model so that the domain adaptability of the teacher model exceeds a set threshold.

[0028] In one possible implementation, the training of the detector based on the first classification result and the second classification result to obtain the target detection model includes:

[0029] Using the teacher model to perform classification processing on the training images of the target domain at a fully connected layer, and using the classification processing results as pseudo labels of the target domain to supervise the first classification results and the second classification results of the student model;

[0030] A source domain label is obtained, and the source domain label is used to supervise the first classification result of the student model.

[0031] In one possible implementation, obtaining instance-level features of objects contained in a training image based on a detector includes:

[0032] Obtaining original image features and candidate frames of the training image based on a detector backbone network;

[0033] Based on the original image features, instance-level features corresponding to the candidate boxes are obtained through a pooling operation.

[0034] A second aspect of the present application provides a training device for an object detection model, comprising:

[0035] A feature acquisition module is used to obtain instance-level features of the objects contained in the training image based on the detector;

[0036] A first classification module is configured to perform a fully connected layer classification process based on the instance-level features to obtain a first classification result;

[0037] A second classification module is configured to perform prototype classification processing based on the instance-level features and the first classification result to obtain a second classification result;

[0038] A model training module is used to train the detector based on the first classification result and the second classification result to obtain a target detection model.

[0039] Through the above technical solutions, it can be seen that the embodiment of the present application discloses a method and device for training a target detection model, the method comprising: obtaining instance-level features of the target contained in the training image based on the detector; performing fully connected layer classification processing based on the instance-level features to obtain a first classification result; performing prototype classification processing based on the instance-level features and the first classification result to obtain a second classification result; training the detector based on the first classification result and the second classification result to obtain a target detection model. The above solution adds prototype classification processing on the basis of fully connected layer classification, so that feature alignment and self-training based on pseudo-labels can fully interact; in the implementation of the solution, the instance feature distributions of the source domain and the target domain can be aligned, so that the target domain prototype generated can well reflect the target domain feature distribution, and the classification based on the class prototype further regularizes the target domain distribution, thereby helping the model to cluster similar instance features. Therefore, the model can learn better pseudo-labels based on the clustered features, helping the model to be better trained, so that the detection model finally trained can have better domain adaptability when performing target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0041] Figure 1 This is an example diagram of "domain offset" in domain adaptive object detection disclosed in an embodiment of this application;

[0042] Figure 2 A flowchart of a method for training a target detection model disclosed in an embodiment of the present application;

[0043] Figure 3 This is a flowchart of obtaining the second classification result disclosed in the embodiment of this application;

[0044] Figure 4 This is a schematic diagram of the algorithm principle of the domain adaptive detection framework based on prototype and fully connected dual classification disclosed in the embodiments of this application;

[0045] Figure 5 A schematic diagram of the structure of a training device for a target detection model disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0046] For the purpose of reference and clarity, the following technical terms, abbreviations or abbreviations are summarized as follows:

[0047] Faster-RCNN: Faster R-CNN is a deep neural network architecture for object detection. It belongs to the two-stage target detection framework. It performs target detection and feature extraction simultaneously in a single neural network, achieving multi-task learning.

[0048] RPN: RPN (Region Proposal Network) is an important component of the object detection algorithm, mainly used to generate candidate regions. RPN generates region proposals on the feature map output by the last layer of the shared convolutional network through a sliding window. The input of RPN is an n×n sliding window of the convolutional feature map. For each sliding window, it predicts object region proposals for k anchor points, each with a corresponding scale.

[0049] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0050] In order to better understand the technical solution of this application, some domain adaptive object detection related solutions in the field are first introduced and explained.

[0051] Feature alignment based on adversarial learning: This type of method primarily uses a domain discriminator and an object detection model for adversarial training. The training process follows feature learning based on gradient backpropagation layers. Specifically, the domain discriminator is used to distinguish between source and target domain features, while the object detector is trained to produce features that fool the domain discriminator. Through training, invariant features between the source and target domains are obtained. The typical method, DA-Faster, uses the FasterRCNN model as a baseline model to align source and target domain features at the image and instance levels, respectively. It also introduces classifier consistency regularization to constrain the consistency of the outputs of the image-level domain discriminator and the instance-level domain discriminator.

[0052] Pseudo-label self-training: Self-training methods generate pseudo-labels for target domain data using a detector trained on the source domain. These pseudo-labels are then used with the target domain data to retrain the model. Due to the domain offset between the target and source domains, these pseudo-labels are erroneous and noisy, and using them directly for model training can compromise performance. Currently, much work is focused on developing better filtering strategies to address the noise in pseudo-labels. Pseudo-label self-training methods are simple yet highly effective because they directly minimize the prediction loss on the target domain.

[0053] Fusion based on feature alignment and pseudo-labeling: Both feature alignment and pseudo-labeling can solve the domain offset problem in domain adaptive target detection to a certain extent. Therefore, a very natural way is to fuse the alignment and pseudo-labeling strategies to solve the domain adaptation problem. A typical example is the adaptive teacher AT scheme. In current implementations, Faster-RCNN is often used as the baseline of the detection model. Faster-RCNN is a classic two-stage target detection algorithm. In the first stage, the RPN network extracts candidate boxes from the original image. In the second stage, based on the original image features and candidate boxes extracted by the backbone network, the instance-level features corresponding to the candidate boxes are obtained through pooling operations (ROIAlign). After that, the instance-level features are fully connected for classification and regression to obtain the category and fine-tuning box of the instance features.

[0054] Figure 2 This is a flowchart of a method for training a target detection model disclosed in an embodiment of the present application. Figure 2 The training method of the target detection model shown is an improvement to the traditional AT implementation method, and is mainly used in the student model in the AT implementation method. Figure 2 As shown, the training method of the target detection model may include:

[0055] Step 201: Obtain instance-level features of objects contained in a training image based on a detector.

[0056] The detector is a raw, untrained detector comprising a backbone network and a detection head. By processing a training image with the backbone network, candidate boxes for objects identified in the training image can be obtained. In one implementation, obtaining instance-level features of objects contained in the training image based on the detector may include: obtaining original image features and candidate boxes of the training image based on the backbone network of the detector; and obtaining instance-level features corresponding to the candidate boxes through a pooling operation based on the original image features.

[0057] Step 202: Perform fully connected layer classification processing based on the instance-level features to obtain a first classification result.

[0058] In this application, after obtaining instance-level features, two classification processes are required: one is the fully connected layer classification process, and the other is the prototype-based classification process. The first classification result obtained by the fully connected layer classification process includes the type and classification score, which can be understood as the confidence level of the corresponding classification. The classification process of the fully connected layer is the same as the traditional solution and will not be further explained here.

[0059] Step 203: Perform prototype classification processing based on the instance-level features and the first classification result to obtain a second classification result.

[0060] After the fully connected layer classification process is completed, the prototype classification process is performed. In implementation, it is necessary to first construct prototypes of each type, and then determine which type (prototype) the instance-level features should be classified into through corresponding calculations.

[0061] The construction of the prototype is realized based on the first classification result. The specific construction method will be introduced in detail in the embodiment content below and will not be explained in detail here.

[0062] Step 204: Train the detector based on the first classification result and the second classification result to obtain a target detection model.

[0063] In the implementation, after obtaining the first classification result and the second classification result, the first classification result and the second classification result can be supervised based on the pseudo labels and source domain true value labels obtained by the teacher model's classification processing of the fully connected layer of the target domain image, and the relevant classification loss is determined. The overall loss of the detector is finally determined in combination with other losses of the detection model, and the detection model parameters are adjusted based on the overall loss; the above implementation is iteratively performed in the process of continuously inputting training images into the detection model, thereby realizing continuous tuning of the detection model parameters until the adjusted detection model meets the set requirements.

[0064] The traditional AT implementation method is relatively simple in terms of the fusion of feature alignment and pseudo-label classification. It is just a simple fusion, and the feature alignment process and the pseudo-label processing do not fully interact. The inventors of the present application use "prototype" as an effective medium for the interaction between the two methods of feature alignment and pseudo-label processing. The prototype can be calculated using the average value of the same type of instance features, so it can better reflect the distribution of the same type of instance features. In the AT method, "prototype" is used to communicate the feature alignment process and the pseudo-label processing. Specifically, the feature alignment process can align the instance feature distribution of the source domain and the target domain, so that the target domain prototype generated can well reflect the target domain feature distribution, and the prototype-based classification further regularizes the distribution of the target domain, thereby helping the model to cluster similar instance features. Therefore, the model can learn better pseudo-labels based on the clustered features, helping the model to train better.

[0065] The training method of the target detection model described in this embodiment combines prototype classification with traditional fully connected layer classification, and works together in the RCNN stage of the detector training stage (the aforementioned second stage) to further improve the performance of target detection in domain adaptation scenarios.

[0066] In the implementation, before training the target detection model, the parameters of the teacher model and the student model need to be initialized first.

[0067] The training method of the target detection model described in the embodiment of the present application is to train a detection model with domain adaptive capabilities. Since the goal of domain adaptive target detection is to train a detector based on source domain data (labeled) and target domain data (unlabeled), to achieve the best generalization performance on the target domain data. Therefore, in the initial stage of model training, the source domain data can be used to train the student model first, and the training at this stage is a supervised training method. Since the model structure of the student model is exactly the same as that of the teacher model, that is, the parameters are exactly the same. Therefore, after the training is completed, the parameters obtained by the student model training are used as the initialization parameters of the teacher model, so that the teacher model can have a better initialization weight.

[0068] After completing the parameter initialization of the teacher model and the student model, the training scheme of the target detection model described in the present application scheme can be implemented.

[0069] In the above embodiment, the process of obtaining the second classification result can be found in Figure 3 .like Figure 3 As shown, performing prototype classification processing based on the instance-level features and the first classification result to obtain a second classification result may include:

[0070] Step 301: obtaining a classification result in which the classification score of a category in the first classification result reaches a set value, and the classification result is used as a pseudo label set.

[0071] Each result in the first classification includes a category and a classification score, such as "vehicle, 0.8" and "pedestrian, 0.9." Results with too low a classification score can be filtered out, retaining only those with a classification score that meets a set value. This set value can be customized based on needs, such as 0.7. In this case, classification results with a classification score below 0.7 will be deleted, while those with a score above 0.7 will be retained.

[0072] Step 302: Calculate the average of the instance-level features corresponding to all pseudo-labels under each category in the classification result, and use the obtained average as the prototype of the current category.

[0073] After obtaining a set of pseudo-labels, we can further obtain the instance-level features corresponding to each pseudo-label, and then calculate the average of all instance-level features corresponding to each category as the prototype of the current category.

[0074] For prototype initialization, after the teacher model initializes its parameters, we can first use the teacher model to infer the target domain data. The inference set of the target domain images is recorded as P. At this time, the classification score threshold of the pseudo label is set to T, and the pseudo label set can be expressed as:

[0075] P seudo ={p|p∈P,p>T}

[0076] Where P seudo Represents the final pseudo-label set.

[0077] After obtaining the pseudo-label set, the box corresponding to the pseudo-label is ROI Aligned (pooling operation) according to the pseudo-label to obtain the instance-level feature p corresponding to the pseudo-label ins Then, based on the fully connected classification results of instance-level features, the instance-level features are classified (according to the classification scores) and the mean value of each class is obtained. pro ,The process can be expressed as:

[0078]

[0079] In the formula represents the instance-level prototype of the i-th class, and It represents the instance-level feature corresponding to the pseudo-label of category i in the set. N represents the number of pseudo-labels of this category in the set.

[0080] After step 302, proceed to step 303.

[0081] Step 303: Classify the instance-level features of the target contained in the training image based on the prototype to obtain a second classification result.

[0082] Specifically, a prototype-based classification of instance-level features of the objects contained in the training image may be constructed based on the cosine distance to obtain a second classification result.

[0083] This embodiment introduces in detail the construction of the prototype and the specific implementation of obtaining the second classification result, which helps technicians in the field to better understand and implement the technical solution of this application.

[0084] In the aforementioned embodiment, the training of the detector based on the first classification result and the second classification result to obtain the target detection model may include: determining the overall loss of the detector, the overall loss at least including the classification cross entropy loss based on full connection and the cross entropy loss based on prototype classification; adjusting the parameters of the detector based on the overall loss to obtain the target detection model.

[0085] Among them, determining the overall loss of the detector may include: determining a fully connected classification cross entropy loss based on a first classification result of the target instance-level feature; determining a prototype classification-based cross entropy loss based on a second classification result of the target instance-level feature and the corresponding pseudo-label; performing consistency constraints on the first classification result and the second classification result to obtain a constrained loss; and determining the overall loss of the detector based on the fully connected classification cross entropy loss, the prototype classification-based cross entropy loss, the constrained loss, and other detection model-related losses.

[0086] Specifically, after obtaining the prototype of each class in the target domain, in the detector training stage, after a certain instance feature x is given in the RCNN stage, the classification result based on the prototype can be expressed as:

[0087]

[0088] in, Represents the prototype of the jth class, T is the temperature coefficient, sim(x, y) represents the cosine similarity of the feature vectors x and y, based on the calculated prediction result (classification result), according to the pseudo label q of the instance gt , calculate the prototype-based cross entropy loss L pseudo , the calculation process can be expressed as:

[0089]

[0090] Traditional fully connected classification cross entropy loss L fc It can be expressed as:

[0091]

[0092] In the formula Represents the probability that the traditional given x belongs to category i. Therefore, the dual classification loss based on prototype and full connection can be finally expressed as:

[0093] L cls =L cls +L pseudo

[0094] At the same time, for the instance-level features of the target domain, the consistency constraints on the prototype-based and fully connected classification results can be expressed as:

[0095]

[0096] The loss of the entire detector can be expressed as:

[0097]

[0098] Among them L align , L rpn , L reg , They represent the alignment loss of the student model, the loss of the target domain data passing through the student model RPN, the regression loss of the target domain data passing through the student model RCNN stage, and the supervised detection loss generated by the source domain data passing through the student model. These four types of losses are consistent with the AT method.

[0099] Figure 4 This is a schematic diagram of the domain adaptive detection framework algorithm based on prototype and full connection dual classification disclosed in the embodiment of this application, in which the teacher trunk line corresponds to the teacher model and the student trunk line corresponds to the student model. Figure 4 Understand the relevant content of this application. Figure 4 , the detector is trained based on the first classification result and the second classification result to obtain a target detection model, including: using the teacher model to perform classification processing of the training image of the target domain at the fully connected layer, and using the classification processing result as the pseudo label of the target domain to supervise the first classification result and the second classification result of the student model; obtaining the source domain label ( Figure 4 The source domain gt label is used to supervise the first classification result of the student model.

[0100] During the detector training process, the prototype of the corresponding category can be updated according to the newly appeared instance-level features after each iteration. During the model training process, the network parameters are constantly updated, so the mapping of each type of instance in the feature space will also change continuously. Therefore, during the network training process, after each iteration, the prototype needs to be updated with momentum according to the newly appeared instance features. The momentum update process can be expressed as:

[0101]

[0102] In the formula Represents the updated i prototype, represents the i prototype before updating, λ represents the parameter of momentum update, which is 0.99 in this method. represents the instance-level features belonging to class i in the current batch, and M represents the number of instances belonging to class i in the current batch.

[0103] After the training is completed and the target detection model is obtained, the parameters of the target detection model (student model) can be updated to the teacher model so that the domain adaptability of the teacher model exceeds a set threshold.

[0104] The present application provides a training scheme for a domain adaptive target detection model based on dual classification of prototypes and traditional fully connected layers. On top of the AT method, for the prediction boxes output by the teacher model, since each prediction box contains a classification score, a high confidence level is used to filter and obtain a pseudo label, and the instance-level features corresponding to the high confidence level are averaged to obtain the prototype of each category. Afterwards, in the RCNN stage, for each instance feature obtained, not only the traditional fully connected layer is used for classification, but also the cosine distance is used to construct a prototype-based classification, which is also supervised by pseudo labels. In this way, a dual classification framework based on prototypes and traditional full connections is constructed; and in each training iteration, the scheme performs momentum updates on the prototypes of each category based on the positive sample instances (pseudo labels) to ensure the stability and real-time performance of the prototypes.

[0105] In order to verify the effect of the target detection model trained by the present invention on domain adaptive target detection, the inventors conducted relevant detection experiments. Figure 1 As shown in the figure, Cityscapes and FoggyCityscapes are a very classic set of settings in domain adaptation. The images of Cityscapes are street scenes on sunny days, and the images of FoggyCityscapes are street scenes on foggy days. Both are labeled with the same categories, including multiple categories such as cars, bicycles, and people. Cityscapes->FoggyCityscapes means that the source domain is Cityscape and the target domain is FoggyCityscape. It is a classic set of settings for domain adaptation from sunny days to foggy days. The present invention conducts experiments on the Cityscapes->FoggyCityscapes dataset configuration. Figure 1 As shown in the figure, the scene on Cityscapes is a sunny city street scene, which is used as the source domain dataset, while FoggyCityscapes is a foggy city street scene, which is used as the target domain dataset. The evaluation index mAP used in this dataset is the same as the traditional target detection evaluation index:

[0106] mAP (mean Average Precision): It is used to evaluate whether the model can accurately detect the target in the scene and achieve high accuracy and recall by matching the predicted box with the true value box through intersection and union.

[0107] Table 1. Results of Cityscapes->FoggyCityscapes domain adaptation settings

[0108]

[0109] Table 1 shows the accuracy of dual classification based on prototype and full connection and based on the consistency constraints of the two classifications. As shown in Table 1, compared with the baseline, adding prototype classification on the basis of AT can improve the model by 2%, which fully demonstrates that prototype classification can effectively improve the performance of the model, and can effectively complement the original baseline method based on full connection. AT+prototype full connection constraint L1 represents the use of prototype classification, but does not use pseudo labels to directly constrain the prototype classification results. Instead, it is the result obtained by making consistency L1 constraints on the prototype classification and the fully connected classification. Compared with the baseline, the model also has an improvement of 0.7%. When the dual classification method based on prototype and full connection is adopted at the same time, the model achieves the highest performance of 51.4%, which fully demonstrates the effectiveness of the present application solution.

[0110] For the sake of simplicity, the aforementioned method embodiments are described as a series of action combinations. However, those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0111] The above embodiments disclosed in the present application describe the method in detail. The method of the present application can be implemented using various devices. Therefore, the present application also discloses a device, and a specific embodiment is given below for detailed description.

[0112] Figure 5 This is a structural diagram of a training device for a target detection model disclosed in an embodiment of the present application. Figure 5 As shown, the training device 50 of the target detection model may include:

[0113] The feature acquisition module 501 is used to obtain instance-level features of the objects contained in the training image based on the detector.

[0114] The first classification module 502 is configured to perform a fully connected layer classification process based on the instance-level features to obtain a first classification result.

[0115] The second classification module 503 is configured to perform prototype classification processing based on the instance-level features and the first classification result to obtain a second classification result.

[0116] The model training module 504 is used to train the detector based on the first classification result and the second classification result to obtain a target detection model.

[0117] The training scheme of the target detection model described in this embodiment adds prototype classification processing on the basis of fully connected layer classification, so that feature alignment and pseudo-label-based self-training can fully interact; in the implementation of the scheme, the instance feature distributions of the source domain and the target domain can be aligned, so that the target domain prototype generated can well reflect the target domain feature distribution, and the classification based on the class prototype further regularizes the distribution of the target domain, thereby helping the model to cluster similar instance features. Therefore, the model can learn better pseudo-labels based on the clustered features, helping the model to be better trained, so that the detection model finally trained can have better domain adaptability when performing target detection.

[0118] In one implementation, the first classification result includes categories and classification scores, and the second classification module may include: a result acquisition module, used to obtain a classification result in which the classification score of the category in the first classification result reaches a set value, and the classification result is used as a pseudo-label set; a prototype determination module, used to average the instance-level features corresponding to all pseudo-labels under each category in the classification result, and use the obtained average as the prototype of the current category; a classification processing module, used to classify the instance-level features of the target contained in the training image based on the prototype to obtain a second classification result.

[0119] In one implementation, the classification processing module may be specifically configured to construct a prototype-based classification of instance-level features of the target contained in the training image based on the cosine distance to obtain a second classification result.

[0120] In one implementation, the model training module may be further configured to: during the training of the detector, update the prototype of the corresponding category according to the newly appeared instance-level features after each iteration.

[0121] In one implementation, the model training module may include: a loss determination module for determining the overall loss of the detector, wherein the overall loss includes at least a fully connected classification cross entropy loss and a prototype classification-based cross entropy loss; a parameter adjustment module for adjusting the parameters of the detector based on the overall loss to obtain a target detection model.

[0122] In one implementation, the loss determination module can be specifically used to: determine the fully connected classification cross entropy loss based on the first classification result of the target instance-level feature; determine the prototype classification cross entropy loss based on the second classification result of the target instance-level feature and the corresponding pseudo-label; perform consistency constraints on the first classification result and the second classification result to obtain the constrained loss; determine the overall loss of the detector based on the fully connected classification cross entropy loss, the prototype classification cross entropy loss, the constrained loss and other detection model-related losses.

[0123] In one implementation, the target detection model is a student model, and the device may further include: a model updating module, which is used to update the parameters of the student model to the teacher model after obtaining the target detection model, so that the domain adaptability of the teacher model exceeds a set threshold.

[0124] In one implementation, the model training module can be specifically used to: use the teacher model to perform fully connected layer classification processing on the training images of the target domain, and use the results of the classification processing as pseudo labels of the target domain to supervise the first classification results and second classification results of the student model; obtain the source domain label, and use the source domain label to supervise the first classification result of the student model.

[0125] In one implementation, the feature acquisition module can be specifically used to: obtain the original image features and candidate boxes of the training image based on the backbone network of the detector; and obtain instance-level features corresponding to the candidate boxes through a pooling operation based on the original image features.

[0126] The specific implementation of the training device of the target detection model and the various modules it contains, as well as other possible implementations, can be found in the corresponding parts of the method embodiment, and will not be repeated here.

[0127] The training device of any target detection model described in the above embodiments includes a processor and a memory. The feature acquisition module, the first classification module, the second classification module, the model training module, etc. in the above embodiments are all stored in the memory as program modules, and the processor executes the above program modules stored in the memory to realize the corresponding functions.

[0128] The processor contains a kernel, which retrieves the corresponding program module from the memory. There can be one or more kernels, and the kernel parameters can be adjusted to process the access data.

[0129] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0130] In an exemplary embodiment, a computer-readable storage medium is also provided, which can be directly loaded into the internal memory of a computer and contains software code. After being loaded and executed by a computer, the computer program can implement the steps shown in any embodiment of the training method of the above-mentioned target detection model.

[0131] In an exemplary embodiment, a computer program product is also provided, which can be directly loaded into the internal memory of a computer and contains software code. After being loaded and executed by a computer, the computer program can implement the steps shown in any embodiment of the training method of the target detection model described above.

[0132] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0133] It should also be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0134] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0135] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for training a target detection model, characterized in that: include: Obtain instance-level features of the objects contained in the training image based on the detector; Performing a fully connected layer classification process based on the instance-level features to obtain a first classification result; Performing prototype classification processing based on the instance-level features and the first classification result to obtain a second classification result; The detector is trained based on the first classification result and the second classification result to obtain a target detection model.

2. The target detection model training method according to claim 1, characterized in that: The first classification result includes a category and a classification score. The prototype classification process is performed based on the instance-level features and the first classification result to obtain a second classification result, including: Obtaining a classification result in which the classification score of a category in the first classification result reaches a set value, and using the classification result as a pseudo label set; Calculate the average of the instance-level features corresponding to all pseudo labels under each category in the classification result, and use the obtained average as the prototype of the current category; Classify the instance-level features of the target contained in the training image based on the prototype to obtain a second classification result.

3. The target detection model training method according to claim 2, characterized in that: The classifying the instance-level features of the target contained in the training image based on the prototype to obtain a second classification result includes: The instance-level features of the objects contained in the training image are constructed based on the cosine distance to perform prototype-based classification, thereby obtaining a second classification result.

4. The method for training a target detection model according to claim 2, wherein: Also includes: During the training of the detector, the prototype of the corresponding category is updated according to the newly appeared instance-level features after each iteration.

5. The method for training a target detection model according to claim 1, wherein: The step of training the detector based on the first classification result and the second classification result to obtain a target detection model includes: Determining an overall loss of the detector, the overall loss comprising at least a fully connected classification cross entropy loss and a prototype classification cross entropy loss; The detector is adjusted based on the overall loss to obtain a target detection model.

6. The method for training a target detection model according to claim 5, wherein: Determining the overall loss of the detector includes: Determine the fully connected categorical cross entropy loss based on the first classification result of the target instance-level features; Determine the cross entropy loss based on the prototype classification based on the second classification result of the target instance-level features and the corresponding pseudo-label; Performing consistency constraints on the first classification result and the second classification result to obtain a constraint loss; The overall loss of the detector is determined based on the fully connected classification cross entropy loss, the prototype classification based cross entropy loss, the constraint loss and other detection model related losses.

7. The method for training a target detection model according to claim 1, wherein: The target detection model is a student model. After obtaining the target detection model, the method further includes: The parameters of the student model are updated to the teacher model so that the domain adaptability of the teacher model exceeds a set threshold.

8. The method for training a target detection model according to claim 7, wherein: The step of training the detector based on the first classification result and the second classification result to obtain a target detection model includes: Using the teacher model to perform classification processing on the training images of the target domain at a fully connected layer, and using the classification processing results as pseudo labels of the target domain to supervise the first classification results and the second classification results of the student model; A source domain label is obtained, and the source domain label is used to supervise the first classification result of the student model.

9. The method for training a target detection model according to claim 1, wherein: The step of obtaining instance-level features of objects contained in the training image based on the detector includes: Obtaining original image features and candidate frames of the training image based on a detector backbone network; Based on the original image features, instance-level features corresponding to the candidate boxes are obtained through a pooling operation.

10. A training device for a target detection model, characterized in that: include: A feature acquisition module is used to obtain instance-level features of the objects contained in the training image based on the detector; A first classification module is configured to perform a fully connected layer classification process based on the instance-level features to obtain a first classification result; A second classification module is configured to perform prototype classification processing based on the instance-level features and the first classification result to obtain a second classification result; A model training module is used to train the detector based on the first classification result and the second classification result to obtain a target detection model.

Images