Unsupervised model selection method for adaptive model in target detection field

Abstract

The invention belongs to the technical field of target detection, relates to domain adaptive target detection (DAOD), and particularly provides an unsupervised model selection method for a target detection domain adaptive model, which is used for realizing effective evaluation and selection of a target detection model under the condition of not depending on a target domain label so as to realize domain adaptive target detection. According to the method, a combined index based on model variability and domain distribution distance is proposed, in the aspect of model variability, a smooth index score (FIS) is designed, and the flatness of a model in a parameter space is measured through the similarity of target prediction results before and after model disturbance; in the field distribution distance aspect, proposing a prototype distance ratio score (PDR), and measuring the field distribution distance by calculating the distance ratio of different types of targets in the feature space; and finally, normalizing the smoothing index score and the prototype distance ratio score, and combining the normalized smoothing index score and the normalized prototype distance ratio score as a combined index to complete an unsupervised model selection task of the adaptive model in the target detection field.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection, relates to domain adaptive object detection (Domain Adaptive Object Detection, DAOD), and specifically provides an unsupervised model selection method for a domain adaptive model of target detection. Background Art

[0002] Object detection is an important task in computer vision, which involves identifying the categories and locations of multiple objects in images or videos. In recent years, it has shown great potential in many downstream tasks such as autonomous driving, video understanding, and robot navigation. When there are significant domain differences between the training set (source domain) and the test set (target domain) of the object detection network, especially when facing unknown domains, the performance of the object detection model usually decreases significantly. This problem has given rise to the research on Domain Adaptive Object Detection (DAOD), which aims to transfer the labeled source domain detection knowledge to the unlabeled target domain through methods such as adversarial training or pseudo-labeling.

[0003] Existing domain adaptive object detection methods can be mainly divided into two categories: domain alignment and self-training. Domain alignment methods minimize the feature distribution differences between the source domain and the target domain through adversarial training, graph matching, prototype alignment, etc. For example, the DA Faster R-CNN model (DAF) based on adversarial training aligns features by establishing an adversarial discriminator between the source domain and the target domain. The prototype alignment method (GPA) further improves the detection performance of the model in the target domain by aligning feature prototypes of different categories. Self-training methods generate pseudo labels and use the paradigm of teacher network and student network for training. For example, the Mean Teacher (MT) method uses the teacher network to generate pseudo labels of the target domain and guides the learning of the student network through these pseudo labels. The MeanTeacher (CMT) method further introduces contrastive learning to improve the domain adaptability of the model by maximizing the similarity of features of the same type and the difference of features of different types. These methods solve the problem of domain differences to a certain extent, but when measuring the generalization performance of the network in the target domain, they often rely on the labels of the target domain. Labeling in the target domain is often costly and time-consuming, and is not realistic in practical applications.

[0004] In the field of unsupervised model selection (UMS), existing unsupervised model selection methods are mostly based on classification tasks, such as prediction score (PS), average threshold confidence (ATC), entropy score (ES), transfer score (TS) and DEV. The prediction score (PS) and average threshold confidence (ATC) use the prediction confidence of the model to indirectly evaluate the generalization ability of the model. However, since the detection task requires not only classification but also precise positioning of the target position, they cannot be directly applied to the target detection task. The entropy score (ES) indirectly estimates the generalization ability of the model by evaluating the entropy value of the model classification prediction result, but it also faces the problem of being difficult to effectively apply to target detection. The DEV method evaluates the model performance by embedding the adapted feature representation and reducing the target risk during the verification process, but its direct applicability to the target detection task is insufficient. The transfer score (TS) evaluates the model by checking the spatial consistency of the classifier and the transferability and distinguishability of the deep representation. However, in the target detection task, the additional target positioning requirement makes the TS method difficult to directly apply. In addition, the bounding box stability (BoS) method judges the generalization ability of the target detection model by evaluating the stability of the bounding box predicted by the model when the feature is lost. However, the BoS method does not consider the distribution difference between the source domain and the target domain, which makes it limited in practical application.

[0005] In summary, the existing domain adaptive target detection model evaluation methods and unsupervised model selection methods have certain limitations, and cannot effectively evaluate and select the optimal target detection model without target domain labels; therefore, the present invention proposes a new unsupervised model selection method, which can effectively evaluate and select models without relying on target domain labels, and has important research significance and application value. Summary of the invention

[0006] The purpose of the present invention is to provide an unsupervised model selection method for a domain adaptive model of target detection, so as to realize effective evaluation and selection of target detection models without relying on target domain labels, and then realize domain adaptive target detection. The present invention proposes a combined index based on model variability and domain distribution distance. In terms of model variability, a smoothness index score (FIS) is designed to measure the flatness of the model in the parameter space by the similarity of the target prediction results before and after the model perturbation; in terms of domain distribution distance, a prototype distance ratio score (PDR) is proposed to measure the distance of domain distribution by calculating the distance ratio of different categories of targets in the feature space; finally, the smoothness index score (FIS) and the prototype distance ratio score (PDR) are normalized and combined as a combined index to complete the unsupervised model selection task of the domain adaptive model of target detection.

[0007] To achieve the above object, the technical solution adopted by the present invention is:

[0008] An unsupervised model selection method for an adaptive model in the field of target detection, characterized in that it comprises the following steps:

[0009] Step 1. Take any domain adaptive object detection (DAOD) algorithm, perform unsupervised domain adaptation (UDA) training given the source domain and target domain datasets, and obtain T checkpoints during the training process;

[0010] Step 2. For each checkpoint, calculate the smoothness index score (FIS);

[0011] Step 3. For each checkpoint, calculate the prototype distance ratio score (PDR);

[0012] Step 4. Normalize the smoothness index score (FIS) and prototype distance ratio score (PDR) of all checkpoints respectively, and perform data fusion on the normalized results to obtain the detection adaptation score (DAS) of each checkpoint;

[0013] Step 5. Select the checkpoint with the largest detection adaptation score as the model selection result to complete the unsupervised model selection.

[0014] Furthermore, in step 2, the calculation process of the smoothing index score is:

[0015] The original model is represented as f(·; θ), θ represents the parameters of the original model, and γ represents the radius of the parameter space. A perturbation Δ is added to the parameters θ of the original model to obtain the neighborhood model f(·; θ′), θ′ represents the parameters of the neighborhood model, and is represented as θ′←θ+Δ, and ‖Δ||=γ;

[0016] Given an unlabeled image x in the target domaini , the target group predicted by the original model j represents the target number predicted by the original model, b j With p j Respectively represent the target box and classification probability vector of the jth target predicted by the original model, n i is the number of targets predicted by the original model; the target group predicted by the neighborhood model j′ represents the target sequence number predicted by the neighborhood model, and Respectively represent the target box and classification probability vector of the j′th target predicted by the neighborhood model, n′ i is the number of targets predicted by the neighborhood model; calculate the prediction similarity d between the jth predicted target of the original model and the j′th predicted target of the neighborhood model after perturbation jj′ :

[0017]

[0018] Among them, IoU represents the intersection-over-union ratio between prediction boxes, and KL represents the KL divergence between classification probability vectors;

[0019] Then the smoothness index score FIS is calculated as:

[0020]

[0021] Among them, n″ i =min{n i , n′ i}, represents the target domain distribution, σ(j) represents the matching target number of the j-th prediction target of the original model in the neighborhood model prediction target, σ(j′) represents the matching target number of the j′th prediction target of the neighborhood model in the original model prediction target, and σ represents the Hungarian matching algorithm.

[0022] Furthermore, in step 3, the calculation process of the prototype distance ratio score is:

[0023] Set F i Represents the instance feature matrix obtained from the i-th image, and the classification probability of the k-th target is r represents the feature number, represents the number of instance features of the i-th image, Represents the probability that the rth feature of the i-th image belongs to the k-th target;

[0024] In the target domain, calculate the prototype of the k-th target

[0025]

[0026] in, represents the prototype of the k-th target in the target domain, F ir represents the rth instance feature vector of the ith image, represents the target domain distribution;

[0027] In the source domain, calculate the prototype of the k-th target

[0028]

[0029] in, represents the prototype of the k-th target in the source domain, represents the source domain distribution;

[0030] According to the prototype prototype Corresponding to the prototype matrix of the source domain and the target domain Calculate the intra-class prototype distance d intra :

[0031]

[0032] Where K is the number of classification types of the model, M(·,·) represents the L2 distance matrix of each category prototype between prototype matrices, whose size is K×K, and tr(·) represents the trace of the matrix;

[0033] Calculate the prototype distance d between classes inter :

[0034]

[0035] in, M kk′ Represents the distance value of the kth row and k'th column in the L2 distance matrix M;

[0036] According to d intra and d inter Calculate the prototype distance ratio score PDR:

[0037]

[0038] Furthermore, in step 4, the normalization process uses min-max normalization.

[0039] Furthermore, in step 4, the calculation process of the detection adaptation score is:

[0040]

[0041] Where t represents the checkpoint number, DAS t represents the detection adaptation score of the T-th checkpoint, represents the normalized result value of the smoothing index score at the t-th checkpoint, represents the normalized result value of the prototype distance ratio score of the t-th checkpoint, and λ represents the weighting coefficient.

[0042] Based on the above technical solution, the beneficial effects of the present invention are:

[0043] The present invention provides an unsupervised model selection method for a domain adaptive model for target detection. In the process of training a domain adaptive target detection network, the method supports selecting a model checkpoint with good generalization performance on a target domain in an unsupervised manner without any target domain verification set labeling information, that is, the method realizes the unsupervised model selection of the domain adaptive model for target detection without relying on the target domain label, further reduces the manual labeling cost and the model checkpoint selection cost in the process of training and deploying the domain adaptive target detection model, improves the average performance of the selected model, and reduces the practical application threshold of the domain adaptive target detection method. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical scheme and beneficial effects of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments.

[0045] The present invention provides an unsupervised model selection method for a domain adaptive model of target detection, which is used for selecting a model checkpoint with good generalization performance on a target domain in an unsupervised manner during a model training process of a domain adaptive target detection task. The present invention comprises two core scoring indicators: a flatness index score (FIS) and a prototype distance ratio score (PDR), wherein the flatness index score is used to evaluate the flatness of the model in a parameter space, and the prototype distance ratio score is used to evaluate the cross-domain migration capability of the model and the distinguishability of different target types. The two are combined as a detection adaptation score (DAS), which supports the unsupervised screening of the domain adaptive model of target detection.

[0046] In terms of working principle, since the target domain label cannot be obtained for model evaluation in the domain adaptive target detection scenario, the flat minimum of the model cannot be found. Therefore, the present invention derives the following generalization error bound:

[0047] Given an arbitrary constant δ ≥ 0, there exists the hypothesis Among them, h is the hypothesis, is the hypothesis set, θ h represents the parameters of hypothesis h, given any hypothesis ||θ h′ -θh ||2≤τ,θ h′ Assume that h′ is a parameter, τ>0 is a constant, and h′ is located in the parameter neighborhood of h with radius τ, then the following generalization error bound holds with probability at least 1-δ:

[0048]

[0049] in, represents the source domain, represents the target domain, represents the risk of hypothesis h′ in the target domain, represents the risk of hypothesis h in the target domain, represents the risk of hypothesis h in the source domain, Source Domain and target domain The distribution difference between them, Ω is a constant term;

[0050] The above generalization error bound shows that, in addition to the constant Ω, the risk of hypothesis h′ in the target domain within the neighborhood of hypothesis h with a radius of τ can be defined by three terms: flatness (the prediction difference between the original hypothesis and the neighborhood hypothesis), the error in the source domain, and the distribution difference between the source domain and the target domain; usually, the error in the source domain can be minimized by labeled source domain samples, so the flat minimum can be found by minimizing the first and third terms; flatness can be regarded as the variability of the model (the difference between the predictions of the model output before and after perturbing the model parameters), and the minimum depends on the domain distribution distance in domain adaptive target detection, so the present invention proposes a smoothness index score (FIS) to evaluate flatness by measuring the fluctuations in classification and positioning before and after the parameter perturbation of the model, and also proposes a prototype distance ratio score (PDR) to find the minimum by measuring the transferability and distinguishability of the model.

[0051] The technical solutions and beneficial effects of the present invention are further described in detail below in conjunction with exemplary embodiments.

[0052] This embodiment provides an unsupervised model selection method for an adaptive model in the field of target detection, comprising the following steps:

[0053] Step 1. Take any domain adaptive object detection (DAOD) algorithm, perform unsupervised domain adaptation (UDA) training given the source domain and target domain datasets, and obtain f checkpoints during the training process;

[0054] Step 2. For each checkpoint, calculate the smoothness index score (FIS);

[0055] The smoothness index score (FIS) evaluates the flatness of the model in the parameter space by measuring the prediction difference before and after the model parameters are perturbed. For the target detection task, it calculates the difference between the two prediction branches of classification and localization. Specifically, the original model is represented as f(·; θ), θ represents the parameters of the original model, and γ represents the radius of the parameter space. A perturbation Δ is added to the parameters θ of the given original model to obtain the neighborhood model f(·; θ′), θ′ represents the parameters of the neighborhood model, that is: θ′←θ+Δ, and the perturbed γ is controlled to be a constant, ||Δ||=γ, so the neighborhood model f(·; θ′) is located on the hypersphere of fixed radius of the original model f(·; θ); The correspondence between the original model and the neighborhood model in the target detection prediction results is measured. Given an unlabeled image x in the target domain of the input model, i , the original model will predict a set of targets j represents the target number predicted by the original model, b j With p j Respectively represent the target box and classification probability vector of the jth target predicted by the original model, n i The number of targets predicted by the original model; correspondingly, the neighborhood model obtains a set of predictions for another set of targets j′ represents the target sequence number predicted by the neighborhood model, and Respectively represent the target box and classification probability vector of the j′th target predicted by the neighborhood model, n′ i The number of targets predicted for the neighborhood model;

[0056] Use d jj′ (f γ (x i ; θ)) means in the same image x i The predicted similarity between the original model before perturbation and the neighborhood model after perturbation for the jth predicted target in the image. The smaller the calculated value, the smaller the difference between the two predicted targets. jj′ Specifically expressed as:

[0057]

[0058] Among them, IoU represents the intersection-over-union ratio between prediction boxes, and KL represents the KL divergence between classification probability vectors;

[0059] On this basis, the smoothness index score (FIS) of the model parameter θ in the parameter space radius γ is calculated as follows:

[0060]

[0061] Among them, n″ i =min{n i , n′i}, indicating the model's performance on the input image x before and after perturbation i The minimum number of predicted targets; represents the target domain distribution, σ(j) represents the matching target number of the jth prediction target of the original model (model before perturbation) in the prediction target of the neighborhood model (model after perturbation), and the matching process is performed using the Hungarian algorithm, that is, σ represents the Hungarian algorithm; similarly, σ(j′) represents the matching target number of the j′th prediction target of the neighborhood model (model after perturbation) in the prediction target of the original model (model before perturbation);

[0062] Step 3. For each checkpoint, calculate the prototype distance ratio score (PDR);

[0063] The prototype distance ratio score (PDR) is used to effectively evaluate the transferability and distinguishability of the domain adaptive object detection model. The prototype is calculated based on instance features, such as the proposal feature in Faster RCNN. Specifically, the feature F is set i Represents the matrix composed of all instance features of the i-th image in the dataset, and the classification probability of the k-th target is r represents the proposal feature number, Represents the number of proposal features of the i-th image, Indicates the probability that the r-th proposal feature of the i-th image belongs to the k-th target;

[0064] In the target domain, the prototype of the k-th target Calculated by the following formula:

[0065]

[0066] in, represents the prototype of the k-th target in the target domain, d represents the number of channels of the proposal feature, F ir represents the rth proposa1 feature vector of the ith image, represents the target domain distribution;

[0067] Similarly, in the source domain, the prototype of the k-th target Calculated by the following formula:

[0068]

[0069] in, represents the prototype of the k-th target in the source domain, represents the source domain distribution;

[0070] Thus, the prototype matrix of the source domain and the target domain by category is obtained K is the number of classification types of the target detection network; further calculation of the intra-class prototype distance d intra , expressed as:

[0071]

[0072] Where M(·,·) represents the L2 distance matrix between prototype matrices by category, whose size is K×K, and tr(·) represents the trace of the matrix;

[0073] Calculate the prototype distance d between classes inter , expressed as:

[0074]

[0075] in, M kk′ represents the distance value of the kth row and kth column in the L2 distance matrix M; d intra and d inter Combined, we get the Prototype Distance Ratio score (PDR):

[0076]

[0077] Step 4. Normalize the smoothness index score (FIS) and prototype distance ratio score (PDR) of all checkpoints respectively, and perform data fusion on the normalized results to obtain the detection adaptation score (DAS) of each checkpoint;

[0078] At different checkpoints in a model training, the smoothness index score (FIS) and prototype distance ratio score (PDR) of each checkpoint are calculated, and the smoothness index score (FIS) and prototype distance ratio score (PDR) of all checkpoints are min-max normalized respectively. The normalized results are fused to obtain the detection adaptation score (DAS) of each checkpoint, which is specifically expressed as:

[0079]

[0080] Where t represents the checkpoint number, DAS t represents the detection adaptation score of the t-th checkpoint, represents the normalized result value of the smoothing index score at the t-th checkpoint, represents the normalized result value of the prototype distance ratio score of the t-th checkpoint; λ represents the weighting coefficient, which is used to balance the contribution of the two indicators. In this embodiment, the value is 1;

[0081] Step 5. Select the checkpoint with the largest detection adaptation score as the model selection result to complete the unsupervised model selection.

[0082] To verify the effectiveness of the present invention, three existing classic target detection field adaptation evaluation data are set: Real-to-Art (Pascal VOC as the source domain and Cityscapes as the target domain), Weather Adaptation (Weather Adaptation, Cityscapes as the source domain and Foggy Cityscapes as the target domain), and Synthetic-to-Real Adaptation (Synthetic-to-Real Adaptation, Sim10k as the source domain and Cityscapes as the target domain). The detection adaptation score proposed in the present invention is used on four classic target detection field adaptation models, and the classic target detection field adaptation models are DA Faster R-CNN (DAF), MeanTeacher (MT), Adaptive Teacher (AT), and ContrastiveMean Teacher (CMT). For a model training process in a data setting, the true detection performance index of the last model checkpoint (i.e., the true performance of the checkpoint on the target domain verification set under the data setting) and the best model checkpoint in the training process are taken, and compared with the true detection performance index of the checkpoint selected by the highest detection adaptation score. The comparison results are shown in Table 1. It can be seen from the table that the detection adaptation score (DAS) proposed in the present invention can stably select a checkpoint with a performance not lower than the last checkpoint in the training process, and the model performance selected by the detection adaptation score proposed in the present invention can be close to the performance of the best checkpoint in each training process.

[0083] Table 1

[0084]

[0085] In order to verify the superiority of the present invention over other related model selection methods, the PS, ES, ATC (thresholds are 0.3, 0.4, 0.95), FD, and BoS methods are used to score the above three data settings and four domain adaptive target detection models, and compared with the actual detection performance of the model checkpoints selected by the present invention. The comparison results are shown in Table 2. It can be seen from the table that the detection adaptation score proposed by the present invention exceeds other related model selection methods, with an AP50 mean of 42.52, which exceeds the last checkpoint by 3.25 average percentage points, as well as other model selection methods, thereby proving the effectiveness and superiority of the present invention in model selection effect.

[0086] Table 2

[0087]

[0088] The above description is only a specific implementation mode of the present invention. Any feature disclosed in this specification, unless otherwise stated, can be replaced by other alternative features that are equivalent or have similar purposes; all the disclosed features, or all the steps in the methods or processes, except for mutually exclusive features and / or steps, can be combined in any way.

Claims

1. An unsupervised model selection method for adaptive models in the field of target detection, characterized in that: The following steps are involved: Step 1. Take any domain adaptive object detection algorithm, perform unsupervised domain adaptive training given the source domain and target domain datasets, and obtain T checkpoints during the training process; Step 2. For each checkpoint, calculate the smoothness index score (FIS); Step 3. For each checkpoint, calculate the prototype distance ratio score (PDR); Step 4. Normalize the smoothness index score (FIS) and prototype distance ratio score (PDR) of all checkpoints respectively, and perform data fusion on the normalized results to obtain the detection adaptation score (DAS) of each checkpoint; Step 5. Select the checkpoint with the largest detection adaptation score as the model selection result to complete the unsupervised model selection.

2. The unsupervised model selection method for the target detection domain adaptive model according to claim 1, characterized in that: In step 2, the calculation process of the smoothing index score is: The original model is represented as f(·; θ), θ represents the parameters of the original model, and γ represents the radius of the parameter space. A disturbance Δ is added to the parameters θ of the original model to obtain the neighborhood model f(·; θ′), θ′ represents the parameters of the neighborhood model, and is represented as θ′←θ+Δ, and ||Δ||=γ; Given an unlabeled image x in the target domain i , the target group predicted by the original model j represents the target number predicted by the original model, b j With p j They represent the target box and classification probability vector of the jth target predicted by the original model, n i is the number of targets predicted by the original model; the target group predicted by the neighborhood model j′ represents the target sequence number predicted by the neighborhood model, and Respectively represent the target box and classification probability vector of the j′th target predicted by the neighborhood model, n′ i is the number of targets predicted by the neighborhood model; calculate the prediction similarity d between the jth predicted target of the original model and the j′th predicted target of the neighborhood model after perturbation jj′ : Among them, IoU represents the intersection-over-union ratio between prediction boxes, and KL represents the KL divergence between classification probability vectors; Then the smoothness index score FIS is calculated as: Among them, n″ i =min{n i ,n′ i }, represents the target domain distribution, σ(j) represents the matching target number of the j-th prediction target of the original model in the neighborhood model prediction target, σ(j′) represents the matching target number of the j′-th prediction target of the neighborhood model in the original model prediction target, and σ represents the Hungarian algorithm.

3. The unsupervised model selection method for the target detection domain adaptive model according to claim 1 is characterized in that: In step 3, the calculation process of the prototype distance ratio score is: Set F i Represents the instance feature matrix of the i-th image, and the classification probability of the k-th target is r represents the feature number, represents the number of instance features of the i-th image, Represents the probability that the rth instance feature of the i-th image belongs to the k-th target; In the target domain, calculate the prototype of the k-th target in, represents the prototype of the k-th target in the target domain, F ir represents the rth instance feature vector of the ith image, represents the target domain distribution; In the source domain, calculate the prototype of the k-th target in, represents the prototype of the k-th target in the source domain, represents the source domain distribution; According to the prototype prototype Corresponding to the prototype matrix of the source domain and the target domain Calculate the intra-class prototype distance d intra : Where K is the number of classification types of the model, M(·,·) represents the L2 distance matrix between prototype matrices by category, with a size of K×K, and tr(·) represents the trace of the matrix; Calculate the prototype distance d between classes inter : in, M kk′ Represents the distance value of the kth row and k'th column in the L2 distance matrix M; According to d intra and d inter Calculate the prototype distance ratio score PDR:

4. The unsupervised model selection method for the target detection domain adaptive model according to claim 1, characterized in that: In step 4, the normalization process uses min-max normalization.

5. The unsupervised model selection method for the target detection domain adaptive model according to claim 1, characterized in that: In step 4, the calculation process of the detection adaptation score is: Where t represents the checkpoint number, DAS t represents the detection adaptation score of the t-th checkpoint, represents the normalized result value of the smoothing index score at the t-th checkpoint, represents the normalized result value of the prototype distance ratio score of the t-th checkpoint, and λ represents the weighting coefficient.

Images