An Unsupervised Model Selection Method for Adaptive Models in Object Detection

Abstract

This invention belongs to the field of object detection technology, specifically Domain Adaptive Object Detection (DAOD). It provides an unsupervised model selection method for domain adaptive object detection models, enabling effective evaluation and selection of object detection models without relying on target domain labels, thereby achieving domain adaptive object detection. This invention proposes a combined index based on model variability and domain distribution distance. Regarding model variability, a smoothness index score (FIS) is designed, measuring the model's flatness in the parameter space by the similarity of target prediction results before and after model perturbation. Regarding domain distribution distance, a prototype distance ratio score (PDR) is proposed, measuring the distance of the domain distribution by calculating the distance ratio of different target categories in the feature space. Finally, the smoothness index score and the prototype distance ratio score are normalized and combined as a combined index to complete the unsupervised model selection task for domain adaptive object detection models.

Description

Technical Field

[0001] This invention belongs to the field of object detection technology, and relates to Domain Adaptive Object Detection (DAOD). Specifically, it provides an unsupervised model selection method for domain adaptive models in object detection. Background Technology

[0002] Object detection is an important task in computer vision, involving the identification of the category and location of multiple objects in an image or video. In recent years, it has shown great potential in many downstream tasks such as autonomous driving, video understanding, and robot navigation. When there are significant domain differences between the training set (source domain) and the test set (target domain) of the object detection network, especially when facing unknown domains, the performance of the object detection model usually drops significantly. This problem has spurred the research of Domain Adaptive Object Detection (DAOD), which aims to transfer labeled source domain detection knowledge to the unlabeled target domain through adversarial training or pseudo-labels.

[0003] Existing domain-adaptive object detection methods can be broadly categorized into two types: domain alignment and self-training. Domain alignment methods minimize the feature distribution differences between the source and target domains through adversarial training, graph matching, and prototype alignment. For example, the adversarial training-based DA Faster R-CNN (DAF) model aligns features by establishing an adversarial discriminator between the source and target domains. Prototype alignment (GPA) further improves the model's detection performance in the target domain by aligning feature prototypes of different categories. Self-training methods generate pseudo-labels and utilize the paradigm of teacher and student networks for training. For instance, the Mean Teacher (MT) method uses the teacher network to generate pseudo-labels for the target domain and uses these pseudo-labels to guide the learning of the student network. The MeanTeacher (CMT) method further introduces contrastive learning, which improves the model's domain adaptability by maximizing the similarity of features of the same class and the difference of features of different classes. These methods solve the domain difference problem to some extent, but when measuring the generalization performance of the network on the target domain, they often rely on the labels of the target domain. Labeling on the target domain is often costly and time-consuming, and is not practical in real applications.

[0004] In the field of unsupervised model selection (UMS), existing unsupervised model selection methods are mostly based on classification tasks, such as prediction score (PS), average threshold confidence (ATC), entropy score (ES), transfer score (TS), and DEV. Prediction score (PS) and average threshold confidence (ATC) indirectly evaluate a model's generalization ability by using the model's prediction confidence. However, since object detection tasks require not only classification but also precise target location, they cannot be directly applied to object detection tasks. Entropy score (ES) indirectly estimates a model's generalization ability by evaluating the entropy value of the model's classification prediction results, but it also faces the problem of being difficult to apply effectively to object detection. The DEV method evaluates model performance by embedding adapted feature representations and reducing target risk during validation, but its direct applicability to object detection tasks is insufficient. Transfer score (TS) evaluates the model by checking the spatial consistency of the classifier and the transferability and discriminability of the deep representation, but in object detection tasks, the additional requirement of target localization makes the TS method difficult to apply directly. In addition, the bounding box stability (BoS) method judges the generalization ability of an object detection model by evaluating the stability of the bounding boxes predicted by the model in the case of feature loss. However, the BoS method does not consider the distribution differences between the source and target domains, limiting its effectiveness in practical applications.

[0005] In summary, existing evaluation methods for domain-adaptive object detection models and unsupervised model selection methods have certain limitations, failing to effectively evaluate and select the optimal object detection model without target domain labels. Therefore, this invention proposes a novel unsupervised model selection method that can effectively evaluate and select models without relying on target domain labels, which has significant research and application value. Summary of the Invention

[0006] The purpose of this invention is to provide an unsupervised model selection method for adaptive target detection models, enabling effective evaluation and selection of target detection models without relying on target domain labels, thereby achieving domain-adaptive target detection. This invention proposes a combined index based on model variability and domain distribution distance. Regarding model variability, a Smoothing Index Score (FIS) is designed, measuring the flatness of the model in the parameter space by the similarity of target prediction results before and after model perturbation. Regarding domain distribution distance, a Prototype Distance Ratio Score (PDR) is proposed, measuring the distance of the domain distribution by calculating the distance ratio of different target categories in the feature space. Finally, the Smoothing Index Score (FIS) and the Prototype Distance Ratio Score (PDR) are normalized and combined as a combined index to complete the unsupervised model selection task for domain-adaptive target detection models.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] An unsupervised model selection method for adaptive models in the field of object detection, characterized by the following steps:

[0009] Step 1. Adopt the Domain Adaptive Object Detection (DAOD) algorithm and perform unsupervised domain adaptation (UDA) training given the source and target domain datasets, and obtain T checkpoints during the training process;

[0010] Step 2. For each checkpoint, calculate the smoothness index score (FIS);

[0011] Step 3. For each checkpoint, calculate the Prototype Distance Ratio (PDR) score;

[0012] Step 4. Normalize the smoothness index score (FIS) and prototype distance ratio score (PDR) of all checkpoints, and fuse the normalization results to obtain the detection adaptation score (DAS) for each checkpoint.

[0013] Step 5. Select the checkpoint with the highest detection fitness score as the model selection result to complete the unsupervised model selection.

[0014] Furthermore, in step 2, the calculation process for the smoothing index score is as follows:

[0015] The original model is represented as f(·;θ), where θ represents the parameters of the original model. Let γ represent the radius of the parameter space. Add a perturbation Δ to the parameters θ of the original model to obtain the neighborhood model f(·;θ′), where θ′ represents the parameters of the neighborhood model, expressed as θ′←θ+Δ, and ‖Δ||=γ;

[0016] Given an unlabeled image x in the target domaini The target group predicted by the original model j represents the target index predicted by the original model, b j With p j Let n represent the bounding box and classification probability vector of the j-th target predicted by the original model, respectively. i The original model predicts the number of targets; the neighborhood model predicts the target group. j′ represents the target index predicted by the neighborhood model. and Let n' and n' represent the bounding box and classification probability vector of the j′-th target predicted by the neighborhood model, respectively. i The number of targets predicted by the neighborhood model; calculate the prediction similarity d between the j-th predicted target of the original model and the j′ predicted target of the perturbed neighborhood model. jj′ :

[0017]

[0018] Where IoU represents the intersection-union ratio between predicted boxes, and KL represents the KL divergence between classification probability vectors;

[0019] The smoothing index score FIS is then calculated as follows:

[0020]

[0021] Where, n″ i =min{n i , n′ i}, Let σ(j) represent the target domain distribution, σ(j) represent the matching target index of the j-th predicted target in the neighborhood model corresponding to the j-th predicted target in the original model, σ(j′) represent the matching target index of the j′-th predicted target in the neighborhood model corresponding to the j-th predicted target in the original model, and σ represents the Hungarian matching algorithm.

[0022] Furthermore, in step 3, the calculation process for the prototype distance ratio score is as follows:

[0023] Setting F i Let represent the instance feature matrix obtained from the i-th image, where the classification probability of the k-th class object is . r represents the feature number. This represents the number of instance features in the i-th image. This represents the probability that the r-th feature of the i-th image belongs to the k-th class of objects;

[0024] In the target domain, compute the prototype of the k-th type of target.

[0025]

[0026] in, F represents the prototype of the k-th type of target in the target domain. ir This represents the feature vector of the r-th instance of the i-th image. Indicates the distribution of the target domain;

[0027] In the source domain, compute the prototype of the k-th type of target.

[0028]

[0029] in, This represents the prototype of the k-th type of target in the source domain. Indicates the source domain distribution;

[0030] According to the prototype prototype The prototype matrices of the source and target domains are obtained accordingly. Calculate the intra-class prototype distance d intra :

[0031]

[0032] Where K is the number of classification types in the model, M(·,·) represents the L2 distance matrix between each class prototype matrix, and its size is K×K, and tr(·) represents the trace of the matrix.

[0033] Calculate the prototype distance d between classes inter :

[0034]

[0035] in, M kk′ This represents the distance value in the k-th row and k′-th column of the L2 distance matrix M;

[0036] According to d intra and d inter Calculate the Prototype Distance Ratio (PDR) score:

[0037]

[0038] Furthermore, in step 4, the normalization process uses min-max normalization.

[0039] Furthermore, in step 4, the calculation process for the adaptation score is as follows:

[0040]

[0041] Where t represents the checkpoint number, DAS t This represents the detection adaptation score for the T-th checkpoint. This represents the normalized result of the smoothing index score at the t-th checkpoint. λ represents the normalized result of the prototype distance ratio score at the t-th checkpoint, and λ represents the weighting coefficient.

[0042] Based on the above technical solution, the beneficial effects of the present invention are as follows:

[0043] This invention provides an unsupervised model selection method for domain-adaptive object detection models. During the training of a domain-adaptive object detection network, it supports the unsupervised selection of model checkpoints with good generalization performance in the target domain without any target domain validation set annotation information. That is, it realizes the unsupervised model selection of domain-adaptive object detection models without relying on target domain labels, further reducing the manual annotation cost and model checkpoint selection cost in the training and deployment process of domain-adaptive object detection models, improving the average performance of the selected models, and lowering the threshold for practical application of domain-adaptive object detection methods. Detailed Implementation

[0044] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the embodiments.

[0045] This invention provides an unsupervised model selection method for adaptive models in the object detection domain. It is used to select model checkpoints with good generalization performance in the target domain during model training for domain-adaptive object detection tasks in an unsupervised manner. This invention includes two core scoring metrics: the Flatness Index Score (FIS) and the Prototype Distance Ratio (PDR). The FIS assesses the flatness of the model in the parameter space, while the PDR assesses the model's cross-domain transferability and discriminability against different target types. The two metrics are combined to form the Detection Adaptation Score (DAS), supporting unsupervised selection of domain-adaptive models for object detection.

[0046] In terms of working principle, since the target domain label cannot be obtained for model evaluation in the domain-adaptive object detection scenario, the flat minima of the model cannot be found. Therefore, this invention derives the following generalization error bound:

[0047] Given any constant δ≥0, there exists an assumption Where h is an assumption, Let θ be the hypothesis set. h The parameter represents hypothesis h, given any hypothesis. ||θ h′ -θh ||2≤τ,θ h′ Assuming that h′ is a parameter, τ>0 is a constant, and h′ lies within the parameter neighborhood of h with radius τ, then the following generalization error bound holds with a probability of at least 1-δ:

[0048]

[0049] in, Represents the source domain. Indicates the target domain. This indicates the risk of assuming h′ in the target domain. This indicates the risk of hypothesis h in the target domain. This indicates the risk of hypothesis h in the source domain. For source domain and target domain The distributional differences between them, where Ω is a constant term;

[0050] The generalization error bounds described above indicate that, in addition to the constant Ω, the risk of hypothesis h′ in the target domain within the neighborhood of hypothesis h with radius τ can be defined by three terms: flatness (the difference in predictions between the original hypothesis and the neighborhood hypothesis), error in the source domain, and distribution difference between the source and target domains. Typically, the error in the source domain can be minimized using labeled source domain samples; therefore, the minimum of flatness can be found by minimizing the first and third terms. Flatness can be viewed as the variability of the model (the difference between the predictions of the model output before and after perturbing the model parameters), and the minimum depends on the neighborhood distribution distance in neighborhood adaptive object detection. Therefore, this invention proposes the Smoothing Index Score (FIS) to evaluate flatness by measuring the fluctuations in classification and localization before and after perturbing the model parameters, and also proposes the Prototype Distance Ratio Score (PDR) to find the minimum by measuring the transferability and discriminability of the model.

[0051] The technical solution and beneficial effects of the present invention will be further described in detail below with reference to exemplary embodiments.

[0052] This embodiment provides an unsupervised model selection method for adaptive models in the field of object detection, including the following steps:

[0053] Step 1. Adopt the Domain Adaptive Object Detection (DAOD) algorithm, perform unsupervised domain adaptation (UDA) training given the source and target domain datasets, and obtain f checkpoints during the training process;

[0054] Step 2. For each checkpoint, calculate the smoothness index score (FIS);

[0055] The Smoothness Index Score (FIS) evaluates the flatness of a model in the parameter space by measuring the difference in predictions before and after parameter perturbation. For object detection tasks, it calculates the difference between the classification and localization prediction branches. Specifically, the original model is represented as f(·θ), where θ represents the parameters of the original model, and γ represents the radius of the parameter space. A perturbation Δ is added to the parameters θ of the given original model to obtain the neighborhood model f(·θ′), where θ′ represents the parameters of the neighborhood model, i.e., θ′←θ+Δ. The perturbation γ is kept constant, ||Δ||=γ. Therefore, the neighborhood model f(·θ′) lies on a hypersphere with a fixed radius of the original model f(·θ). The correspondence between the original model and the neighborhood model in the object detection prediction results is measured, given an unlabeled image x of the target domain input model. i The original model will predict a set of targets. j represents the target index predicted by the original model, b j With p j Let n represent the bounding box and classification probability vector of the j-th target predicted by the original model, respectively. i The original model predicts the number of targets; correspondingly, the neighborhood model obtains one set of predicted targets and another set of predicted targets. j′ represents the target index predicted by the neighborhood model. and Let n' and n' represent the bounding box and classification probability vector of the j′-th target predicted by the neighborhood model, respectively. i The number of targets predicted by the neighborhood model;

[0056] Use d jj′ (f γ (x i ;θ)) represents the same image x i The prediction similarity between the original model before perturbation and the neighborhood model after perturbation for the j-th predicted target in the image is calculated as follows: the smaller the value, the smaller the difference between the two predicted targets. jj′ Specifically, it is expressed as follows:

[0057]

[0058] Where IoU represents the intersection-union ratio between predicted boxes, and KL represents the KL divergence between classification probability vectors;

[0059] Based on this, the formula for calculating the smoothness exponential score (FIS) of the model parameter θ under a parameter space radius of γ is:

[0060]

[0061] Where, n″ i =min{n i , n′i} represents the model's performance before and after perturbation for the input image x. i The minimum number of predicted targets; Let σ(j) represent the target domain distribution, and let σ(j) represent the matching target index of the j-th predicted target in the neighborhood model (perturbation model) corresponding to the j-th predicted target in the original model (perturbation model). The matching process uses the Hungarian algorithm, i.e., σ represents the Hungarian algorithm. Similarly, σ(j′) represents the matching target index of the j′-th predicted target in the neighborhood model (perturbation model) corresponding to the j′-th predicted target in the original model (perturbation model).

[0062] Step 3. For each checkpoint, calculate the Prototype Distance Ratio (PDR) score;

[0063] The Prototype Distance Ratio (PDR) is used to effectively evaluate the transferability and discriminability of domain-adaptive object detection models. The prototype is calculated based on instance features, such as the proposal features in Faster R-CNN. Specifically, feature F is defined. i Let represent the matrix consisting of the features of all instances of the i-th image in the dataset, and let represent the classification probability of the k-th class of the target. r represents the proposal feature number. This represents the number of proposal features for the i-th image. This represents the probability that the r-th proposal feature of the i-th image belongs to the k-th class of objects;

[0064] In the target domain, the prototype of the k-th type of target Calculated using the following formula:

[0065]

[0066] in, This represents the prototype of the k-th type of target in the target domain. d represents the number of channels in the proposal feature, F ir This represents the r-th proposa1 feature vector of the i-th image. Indicates the distribution of the target domain;

[0067] Similarly, in the source domain, the prototype of the k-th type of target... Calculated using the following formula:

[0068]

[0069] in, This represents the prototype of the k-th type of target in the source domain. Indicates the source domain distribution;

[0070] This yields the category-based prototype matrices of the source and target domains. K represents the number of classification types in the object detection network; further calculations are made for the intra-class prototype distance d. intra , represented as:

[0071]

[0072] Where M(·,·) represents the class-wise L2 distance matrix between prototype matrices, with a size of K×K, and tr(·) represents the trace of the matrix;

[0073] Calculate the prototype distance d between classes inter , represented as:

[0074]

[0075] in, M kk′ This represents the distance value in the k-th row and k′-th column of the L2 distance matrix M; d intra and d inter Combined, we obtain the Prototype Distance Ratio (PDR) score:

[0076]

[0077] Step 4. Normalize the smoothness index score (FIS) and prototype distance ratio score (PDR) of all checkpoints, and fuse the normalization results to obtain the detection adaptation score (DAS) for each checkpoint.

[0078] In a single model training iteration, at different checkpoints, the smoothness exponential score (FIS) and prototype distance ratio score (PDR) are calculated for each checkpoint. Min-max normalization is then applied to the FIS and PDR scores for all checkpoints. The normalized results are then fused to obtain the detection adaptation score (DAS) for each checkpoint, specifically expressed as follows:

[0079]

[0080] Where t represents the checkpoint number, DAS t This represents the detection adaptation score at the t-th checkpoint. This represents the normalized result of the smoothing index score at the t-th checkpoint. λ represents the normalized result of the prototype distance ratio score at the t-th checkpoint; λ represents the weighting coefficient, used to balance the contribution of the two indicators, and is set to 1 in this embodiment.

[0081] Step 5. Select the checkpoint with the highest detection fitness score as the model selection result to complete the unsupervised model selection.

[0082] To verify the effectiveness of this invention, three existing classic object detection adaptive evaluation datasets were used to set up Real-to-Art (Pascal VOC as the source domain and Cityscapes as the target domain), Weather Adaptation (Cityscapes as the source domain and Foggy Cityscapes as the target domain), and Synthetic-to-Real Adaptation (Sim10k as the source domain and Cityscapes as the target domain). The detection adaptation score proposed in this invention was applied to four classic object detection adaptive models, namely DA Faster R-CNN (DAF), MeanTeacher (MT), Adaptive Teacher (AT), and Contrastive Mean Teacher (CMT). For a model training process in a given data setting, the true detection performance index of the last model checkpoint (i.e., the true performance of the checkpoint on the target domain validation set under this data setting), the best model checkpoint during training, and the true detection performance index of the checkpoint selected by the highest detection adaptation score are compared. The comparison results are shown in Table 1. As can be seen from the table, the Detection Adaptation Score (DAS) proposed in this invention can stably select checkpoints with performance no lower than that of the last checkpoint during training. Furthermore, the model performance selected by the Detection Adaptation Score proposed in this invention can approach the performance of the best checkpoint during each training process.

[0083] Table 1

[0084]

[0085] To verify the superiority of this invention over other related model selection methods, PS, ES, ATC (thresholds of 0.3, 0.4, and 0.95), FD, and BoS methods were used to score the above three data settings and four domain adaptive target detection models. The actual detection performance of the model selected by this invention at the checkpoints was compared with that of the model selected by this invention. The comparison results are shown in Table 2. As can be seen from the table, the detection adaptation score proposed by this invention exceeds that of other related model selection methods, with an AP50 mean of 42.52, which is 3.25 percentage points higher than the average of the last checkpoint and other model selection methods. This proves the effectiveness and superiority of this invention in model selection.

[0086] Table 2

[0087]

[0088] The above description is merely a specific embodiment of the present invention. Any feature disclosed in this specification may be replaced by other equivalent or similar features unless otherwise specified. All disclosed features, or steps in all methods or processes, may be combined in any way except for mutually exclusive features and / or steps.

Claims

1. An unsupervised model selection method for a target detection field adaptive model, characterized in that, Comprising the following steps: Step 1. Take any kind of domain adaptive object detection algorithm, do unsupervised domain adaptive training under the given source domain and target domain data sets, and obtain T checkpoints during the training process; Step 2. For each checkpoint, calculate the smoothness index score (FIS); the calculation process of the smoothness index score is: Let the original model be represented as f(·; θ), where θ represents the parameters of the original model, and let γ represent the radius of the parameter space. Add a perturbation Δ to the parameters θ of the original model to obtain a neighborhood model f(·; θ'), where θ' represents the parameters of the neighborhood model, and is represented as θ ′ ← θ + Δ, and ||Δ|| = γ. Given target domain unlabeled image x i , the target set predicted by the original model j represents the target serial number predicted by the original model, b j , and p j , respectively, represent the target box and classification probability vector of the jth target predicted by the original model, n i , and the target set predicted by the neighborhood model j ′ , respectively, represent the target serial number predicted by the neighborhood model, b , and , respectively, represent the target box and classification probability vector of the jth target predicted by the neighborhood model, n i ′ , and the target set predicted by the neighborhood model ′ , the prediction similarity d jj′ between the jth prediction target of the original model and the jth prediction target of the neighborhood model after perturbation is calculated Wherein, IoU represents the intersection over union between the predicted boxes, and KL represents the KL divergence between the classification probability vectors; Then calculate the smoothness index score FIS as: where n" = min{n i , n' i} and n' = max{n i , n ′}. , σ(j) denotes the matching target sequence number of the jth predicted target of the original model corresponding to the jth predicted target of the neighborhood model in the predicted target of the neighborhood model, and σ(j ′ ) denotes the matching target sequence number of the jth predicted target of the neighborhood model corresponding to the jth predicted target of the original model in the predicted target of the original model, and σ denotes the Hungarian algorithm. Step 3. For each checkpoint, calculate the prototype distance ratio score (PDR); the calculation process of the prototype distance ratio score is: Set F i Xi represents an instance feature matrix of the i-th image, and the classification probability of the k-th target is r represents the feature serial number, Xi represents the number of instance features of the i-th image, Xi represents the probability that the r-th instance feature of the i-th image belongs to the k-th target. In the target domain, compute the prototype of the k-th target class wherein, represents a prototype of the kth class of targets in the target domain, F ir represents the rth instance feature vector of the ith image, represents the target domain distribution; In the source domain, compute the prototype of the k-th target wherein, denotes a prototype of the k-th target in the source domain, denotes the source domain distribution; According to the prototype Prototype Corresponding prototype matrices for the source and target domains are obtained Compute intra-class prototype distance d intra : Wherein, K is the number of classification types of the model, M(·,·) represents the L2 distance matrix between the prototype matrices, with a size of KxK, and tr(·) represents the trace of the matrix; Computing inter-class prototype distance d inter : in, M kk′ Represents the k-th row and k-th column in the L2 distance matrix M. ′ The distance value of the column; According to d intra and d inter Calculate prototype distance ratio score PDR: Step 4. Normalize the smoothness index score (FIS) and the prototype distance ratio score (PDR) of all checkpoints respectively, and fuse the normalized results to obtain the detection adaptation score (DAS) of each checkpoint; Step 5. Select the checkpoint with the maximum detection adaptation score as the model selection result, and complete the unsupervised model selection.

2. The method of claim 1, wherein the target detection domain adaptive model is selected from a plurality of unsupervised models. In step 4, the normalization processing adopts min-max normalization.

3. The method of claim 1, wherein the target detection domain adaptive model is selected from a plurality of unsupervised models. In step 4, the calculation process of the detection adaptation score is: where t represents the checkpoint number, DAS t denotes the detection fitness score of the tth checkpoint, denotes the normalized result value of the smoothness index score of the tth checkpoint, denotes the normalized result value of the prototype distance ratio score of the tth checkpoint, and λ represents a weighting coefficient.

Images