Test-time adaptive object detection method based on pre-trained large models

Abstract

The application discloses a kind of test time self-adaptive target detection methods based on pre-training large model, pre-trained text encoder and image encoder are used to extract the features of input text and image respectively, then input pre-training large model to obtain initial target proposal set, after refining initial target proposal set, the visual label of each refined target proposal is generated, then it is spliced with text features to obtain extended text features, the bounding box of new target proposal is obtained by inputting extended text features and original image features into pre-training large model again, the classification probability vector and confidence based on prediction score are calculated, and the final target proposal set is obtained, finally, post-processing is carried out to obtain the final target detection result.The application realizes fine-grained, context-aware visual-language real-time alignment in the test stage, so as to improve the positioning accuracy and classification robustness of target detection.

Description

Technical Field

[0001] This invention belongs to the field of video object detection technology, and more specifically, relates to a test-time adaptive object detection method based on a pre-trained large model. Background Technology

[0002] Open-vocabulary object detection methods utilize pre-trained visual-language models (such as CLIP) to incorporate textual semantics into the detection process, enabling the recognition of objects belonging to categories not encountered during the training phase. Typical methods include GroundingDINO, GLIP, and OWL-ViT. Their common feature is the use of predefined fixed text cues (e.g., "a photo of a [CLASS]") as category semantic representations during inference. These cues are then aligned with the visual features of the image region through a cross-modal matching mechanism, thereby achieving object localization and classification.

[0003] To address the performance degradation caused by domain shifts (such as weather changes, image noise, and style transfer) during the testing phase, the industry has proposed a Test-Time Adaptation (TTA) method, which mainly employs the following strategies: some methods perform post-processing calibration on the classification confidence of the detector output, such as adjusting the temperature coefficient or reweighting logits; others utilize unsupervised consistency loss to fine-tune some parameters of the detector, such as batch normalization statistics or lightweight adaptation layers; still others adopt a teacher-student architecture to iteratively optimize prediction results during testing. However, existing test-time adaptive methods still have the following problems:

[0004] 1) Existing test-time adaptive methods use fixed text prompts during inference, which cannot dynamically adjust semantic representations based on the visual content of specific target instances in the input image. (1) Fixed prompts are predefined during the training phase and do not take into account changes in the distribution of visual features in the test image due to domain shifts (such as fog, rain, low light, and image perturbation); (2) Semantic misalignment between visual features and static text embeddings will significantly reduce the reliability of cross-modal matching; (3) In open-vocabulary detection scenarios, new categories themselves lack strong supervision signals and are more sensitive to the accuracy of semantic guidance. Static prompts are very likely to lead to false detections or false negatives.

[0005] 2) Existing adaptive methods for testing only apply to the end of the detection process and cannot optimize the generation quality of candidate regions (proposals). (1) Most methods only perform post-processing calibration on the classification prediction score (such as temperature scaling and confidence reweighting), and are powerless to address the localization error and recall reduction of the initial proposal; (2) Domain offset often leads to blurred target appearance or reduced contrast, which severely degrades the performance of the proposal generation module; (3) Since the upper limit of detection performance is limited by the quality of the proposal, optimizing classification alone cannot fundamentally improve the overall mAP.

[0006] 3) Existing methods are usually deeply coupled with specific detectors, have poor generality, and are difficult to transfer to other open vocabulary object detection frameworks. (1) Some methods require modification of the internal structure of the model (such as inserting learnable adaptation layers) or rely on specific training mechanisms (such as consistency regularization, teacher-student distillation); (2) Such strategies cannot be applied to closed-source models, inference APIs, or detectors with different architectures (such as switching from Grounding DINO to OWL-ViT); (3) There is a lack of a general test-time adaptive mechanism that does not require retraining, does not require access to gradients, and is decoupled from the model. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide a test-time adaptive object detection method based on a pre-trained large model. During the testing phase, visual markers are dynamically and adaptively generated and concatenated with text features to obtain extended text features, thereby achieving fine-grained, context-aware real-time visual-language alignment, which improves the localization accuracy and classification robustness of object detection.

[0008] To achieve the above-mentioned objectives, the test-time adaptive object detection method based on a pre-trained large model of the present invention includes the following steps:

[0009] S1: Set the input text and image according to actual needs, and use a pre-trained text encoder to extract the text features of the input text. A pre-trained image encoder is used to extract image features from the input image. ,in, Indicates the number of categories contained in the text. Representing feature dimension, Indicates the size of image features;

[0010] S2: Text Features and image features Inputting a pre-trained large model, the initial set of target proposals is obtained. ,in, This indicates the number of target proposals obtained from the initial detection. , Indicates the first The bounding box of an initial target proposal. Indicates the first The initial classification probability vector of each initial target proposal. Indicates the first The initial target proposal belongs to the category The probability, , Indicates the first Confidence prediction scores for each initial target proposal; simultaneously outputting a maximum-scale feature map. , Indicates the size of the maximum scale feature map;

[0011] S3: Visual markers are generated using the following method:

[0012] For the initial target proposal set Refinement yields a set of refinement target proposals. ,in, This indicates the number of target proposals obtained through refinement. , Indicates the first The boundary frame of a refined target proposal, Indicates the first The initial classification probability vector of a refined target proposal. Indicates the first The refined target proposal belongs to the category The probability, Indicates the first Confidence prediction score for a refined target proposal;

[0013] For the collection of refined target proposals Each refinement target proposal, based on the bounding box For the largest scale feature map Perform the ROI Align operation to obtain the corresponding visual markers. ;

[0014] S4: Will A visual marker Text features By concatenating the elements, we obtain extended text features. :

[0015] ;

[0016] S5: The final set of target proposals for detection is generated using the following method:

[0017] S5.1: Extend text features and image features The pre-trained large model is input for secondary detection to obtain the bounding box of the target proposal. , , This indicates the number of target proposals obtained from the second detection;

[0018] S5.2: Based on the bounding box of each target proposal From the maximum scale feature map Extract the corresponding feature sub-map ,in Represents bounding box The dimensions are then used to calculate the extended prediction score using the following formula. :

[0019] ,

[0020] Among them, superscript Indicates transpose. This represents the feature normalization function;

[0021] S5.3: For refining the set of target proposals middle Each refined target proposal is based on its initial classification probability vector. Filter each category The maximum classification probability is denoted as the index of the refined target proposal corresponding to it. Then, the predicted score for each target proposal obtained from the secondary detection is obtained using the following formula. :

[0022] ;

[0023] S5.4: For each target proposal, based on the predicted score Calculate its classification probability vector and confidence level :

[0024] ,

[0025] ,

[0026] in, This represents the softmax function;

[0027] This yields the final set of target proposals. ;

[0028] S6: Set of proposals for the final goal Post-processing is performed to obtain the final target detection result.

[0029] This invention presents a test-time adaptive object detection method based on a pre-trained large model. It employs pre-trained text encoders and image encoders to extract features from the input text and image, respectively. These features are then input into the pre-trained large model to obtain an initial set of object proposals. The initial set is refined to generate visual labels for each refined object proposal, which are then concatenated with the text features to obtain extended text features. These extended text features and the original image features are then input back into the pre-trained large model to obtain new bounding boxes for the object proposals. Based on the prediction score, the classification probability vector and confidence level are calculated to obtain the final set of object proposals. Finally, post-processing is performed to obtain the final object detection result.

[0030] The present invention has the following beneficial effects:

[0031] 1) In the testing phase, this invention generates visual markers for refined target proposals and then fuses them with the original text features to achieve fine-grained, context-aware real-time visual-language alignment, enabling text features to adapt to the input image, thereby improving target detection performance.

[0032] 2) This invention enables cross-modal interaction between the visual and textual features of the target proposal, allowing both the target proposal generation and classification processes to benefit from dynamic semantic guidance, thereby simultaneously improving detection recall and localization accuracy.

[0033] 3) This invention only improves the test inference stage and does not require updating any model parameters. In practical applications, it can be designed as a plug-and-play universal interface, thus seamlessly integrating with various mainstream open vocabulary target detection systems such as Grounding DINO, GLIP, OWL-ViT, and DetCLIP. Attached Figure Description

[0034] Figure 1 This is a flowchart illustrating a specific implementation of the test-time adaptive target detection method based on a pre-trained large model according to the present invention.

[0035] Figure 2 This is an example diagram of the test-time adaptive object detection method based on a pre-trained large model according to the present invention;

[0036] Figure 3 This is a flowchart of the target proposal refinement method based on significant categories in this embodiment;

[0037] Figure 4 This is a flowchart of the process for generating the final set of detection target proposals in this invention. Detailed Implementation

[0038] The specific embodiments of the present invention will now be described with reference to the accompanying drawings to enable those skilled in the art to better understand the invention. It should be particularly noted that in the following description, detailed descriptions of known functions and designs that might obscure the main content of the invention will be omitted here.

[0039] Example

[0040] Figure 1 This is a flowchart illustrating a specific implementation of the test-time adaptive object detection method based on a pre-trained large model according to the present invention. Figure 2 This is an example diagram of the test-time adaptive object detection method based on a pre-trained large model according to the present invention. Figure 1 , Figure 2 As shown, the test-time adaptive object detection method based on a pre-trained large model of the present invention includes the following steps:

[0041] S101: Generate input features:

[0042] The input text and image are set according to actual needs, and a pre-trained text encoder is used to extract the text features of the input text. A pre-trained image encoder is used to extract image features from the input image. ,in, Indicates the number of categories contained in the text. Representing feature dimension, Indicates the size of image features.

[0043] S102: Detect the initial target proposal set:

[0044] Text features and image features Input a pre-trained large model (in this embodiment, the Grounding DINO model is used) to detect the initial set of target proposals. ,in, This indicates the number of target proposals obtained from the initial detection. , Indicates the first The bounding box of each target proposal. Indicates the first The initial classification probability of each target proposal. , Indicates the first The confidence prediction score for each target proposal is generated. Simultaneously, a maximum-scale feature map is output. , This indicates the size of the maximum scale feature map.

[0045] S103: Generate visual markers:

[0046] To optimize the initial detection results, this invention is based on an initial target proposal set. Visual markers were generated, and the specific method is as follows:

[0047] For the initial target proposal set Refinement yields a set of refinement target proposals. ,in, This indicates the number of target proposals obtained through refinement. , Indicates the first A bounding box for a refined target proposal. Indicates the first The initial classification probability vector of a refined target proposal. Indicates the first The refined target proposal belongs to the category The probability, Indicates the first Confidence prediction score for a refined target proposal.

[0048] For the collection of refined target proposals Each refinement target proposal, based on the bounding box For the largest scale feature map Perform the ROI Align operation to obtain the corresponding visual markers. .

[0049] In practical applications, the target proposal refinement method can be set according to actual needs. This embodiment adopts a target proposal refinement method based on salient categories. Figure 3 This is a flowchart of the target proposal refinement method based on salient categories in this embodiment. For example... Figure 3 As shown, the specific steps of the target proposal refinement method based on significant categories in this embodiment include:

[0050] S301: Generate a set of salient categories:

[0051] From the initial target proposal set Choose the one with the highest confidence level The initial target proposal is used as a salient target proposal. The value is determined based on actual needs. For each salient target proposal selected, its initial classification probability vector is used. Determine the category , Indicates the first The original index of each salient target proposal is then used to construct the set of salient categories. .

[0052] S302: Generate a set of preferred target proposals:

[0053] For the initial target proposal set For each initial target proposal, based on the initial classification probability vector Determine the category Filtering and categorizing Belongs to the set of significant categories The initial target proposals constitute the preferred target proposal set. .

[0054] S303: Generate a set of refining target proposals:

[0055] Based on the set of significant categories Select the best target proposal set The target proposal set was obtained by categorizing them. , , Represents the set of salient categories Number of categories. For each target proposal set... The non-maximum suppression (NMS) algorithm is used for refinement to obtain a set of refined target proposals for the corresponding category. .Will A collection of refined target proposals Merge to obtain a refined set of target proposals. Nonmaximum suppression algorithms are commonly used methods for object detection, and their specific processes will not be elaborated here.

[0056] S104: Extended text features:

[0057] Will A visual marker Text features By concatenating the elements, we obtain extended text features. :

[0058] .

[0059] S105: Generate the final set of target proposals for detection:

[0060] Next, a new query selection and decoding process is performed based on the extended text features to generate the final set of target proposals for detection. Figure 4 This is a flowchart illustrating the generation of the final target proposal set in this invention. For example... Figure 4 As shown, the specific method for generating the final target proposal set in this invention includes:

[0061] S401: The bounding box is obtained from the secondary detection:

[0062] Extend text features and image features The pre-trained large model is input for secondary detection to obtain the bounding box of the target proposal. , , This indicates the number of target proposals obtained from the second detection.

[0063] S402: Calculate the extended prediction score:

[0064] Based on the bounding box of each target proposal From the maximum scale feature map Extract the corresponding feature sub-map ,in Represents bounding box The dimensions are then used to calculate the extended prediction score using the following formula. :

[0065] ,

[0066] Among them, superscript Indicates transpose. This represents the feature normalization function.

[0067] S403: Calculate the target proposal prediction score:

[0068] To output the final Class prediction requires class alignment and fusion of the extended prediction scores to obtain the target proposal set. Each target proposal in the category On the predicted score The specific method is as follows:

[0069] For the collection of refined target proposals middle Each refined target proposal is based on its initial classification probability vector. Filter each category The maximum classification probability is denoted as the index of the refined target proposal corresponding to it. Then, the predicted score for each target proposal obtained from the secondary detection is obtained using the following formula. :

[0070] .

[0071] S404: Calculate the classification probability vector and confidence score:

[0072] For each target proposal, based on the predicted score Calculate its classification probability vector and confidence level :

[0073] ,

[0074] ,

[0075] in, This represents the softmax function.

[0076] This yields the final set of target proposals. .

[0077] In practical applications, the final target proposal set can be further refined. The optimization process involves setting a confidence threshold based on actual needs. If the confidence level of the target proposal If the target proposal is selected, it will be retained; otherwise, it will be deleted.

[0078] S106: Post-detection processing:

[0079] Set of proposals for the final goal Post-processing is performed to obtain the final target detection result.

[0080] As described above, this invention, based on a pre-trained text encoder, image encoder, and large model, performs a two-stage optimization process on input text and images that requires no training and only one additional forward inference. In the first stage, standard forward inference is performed on the image to obtain initial detection results, generating visual labels to expand the original text features. In the second stage, forward inference is performed again based on the expanded text features to obtain the final set of target proposals. The entire process is completed during testing without updating model parameters or introducing additional training; both localization accuracy and classification robustness are simultaneously improved simply by dynamically expanding text features.

[0081] To better illustrate the technical effects of this invention, specific examples are used to experimentally verify it. This embodiment evaluates the invention on three mainstream domain-offset open-vocabulary object detection benchmarks—PASCAL-C, COCO-C, and FoggyCityscapes. These datasets cover synthetic image degradation (noise, blur, weather, and 15 other types of perturbations) and real foggy driving scenarios, presenting challenges such as severe appearance distortion and semantic ambiguity, making them suitable for testing the robustness of the model under out-of-distribution conditions. In this embodiment, the invention is based on the pre-trained Grounding DINO framework, with parameters frozen throughout the process, without fine-tuning; the proposed method is applied only during the testing phase. This embodiment uses four existing open-vocabulary object detection models as comparative methods to conduct comparative experiments with this invention, including:

[0082] Grounding DINO, see the literature "Shilong Liu, Zhaoyang Zeng, Tianhe Ren, FengLi, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, etal. Grounding dino: Marrying dino with grounded pre-training for open-setobject detection. In European Conference on Computer Vision, pages 38–55. 620Springer, 2024."

[0083] TDA, see the literature "Adilbek Karmanov, Dayan Guan, Shijian Lu, AbdulmotalebEl Saddik, and Eric Xing. Efficient test-time adaptation of vision-languagemodels. In Proceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition, pages 14162–14171, 2024."

[0084] BCA, see the literature "Lihua Zhou, Mao Ye, Shuaifeng Li, Nianxin Li, Jinlin Wu,Xiatian Zhu, Lei Deng, Hongbin Liu, Jiebo Luo, and Zhen Lei. Bayesian test-time adaptation for object recognition and detection with vision-languagemodels. arXiv preprint arXiv:2510.02750, 2025."

[0085] BCA+, see the document "Lihua Zhou, Mao Ye, Shuaifeng Li, Nianxin Li, XiatianZhu, Lei Deng, Hongbin Liu, and Zhen Lei. Bayesian test-time adaptation for vision-language models. In Proceedings of the IEEE / CVF conference on computervision and pattern recognition, 2025."

[0086] In this embodiment, the evaluation metric uses the standard mAP_{50} (IoU=0.5), and the average is taken across all corruption types or the full validation set. All comparative experiments use the same backbone network (Swin-T / Swin-B) and open vocabulary hints to ensure fairness. Table 1 compares the detection performance of the present invention and the comparative method in this embodiment when implemented using a Swin-T network. Table 2 compares the detection performance of the present invention and the comparative method in this embodiment when implemented using a Swin-B network.

[0087] method PASCAL-C COCO-C FoggyCityscapes Grounding DINO 35.95 21.61 23.88 TDA 37.42 23.00 25.08 BCA 39.10 23.52 26.12 BCA+ 40.67 25.06 26.65 This invention 43.03 26.25 29.10

[0088] Table 1

[0089] method PASCAL-C COCO-C FoggyCityscapes Grounding DINO 63.15 21.61 31.34 TDA 65.64 33.02 34.31 BCA 67.59 41.18 35.47 BCA+ 69.31 44.98 36.22 This invention 71.40 55.17 37.27

[0090] Table 2

[0091] As shown in Tables 1 and 2, this invention achieves superior performance on the mAP metric across all three validation sets. The results demonstrate that this invention, through dynamic visual tagging to enhance text features, effectively mitigates semantic misalignment caused by domain shift and significantly improves detection robustness.

[0092] Although the illustrative specific embodiments of the present invention have been described above to enable those skilled in the art to understand the invention, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of the present invention are protected.

Claims

1. A test-time adaptive object detection method based on a pre-trained large model, characterized in that, Includes the following steps: S1: Set the input text and image according to actual needs, and use a pre-trained text encoder to extract the text features of the input text. A pre-trained image encoder is used to extract image features from the input image. ,in, Indicates the number of categories contained in the text. Representing feature dimension, Indicates the size of image features; S2: Text Features and image features Inputting a pre-trained large model, the initial set of target proposals is obtained. ,in, This represents the number of target proposals obtained from the initial detection. , Indicates the first The bounding box of an initial target proposal. Indicates the first The initial classification probability vector of each initial target proposal. Indicates the first The initial target proposal belongs to the category The probability, , Indicates the first Confidence prediction scores for each initial target proposal; simultaneously outputting a maximum-scale feature map. , Indicates the size of the maximum scale feature map; S3: Visual markers are generated using the following method: For the initial target proposal set Refinement yields a set of refinement target proposals. ,in, This indicates the number of target proposals obtained through refinement. , Indicates the first A bounding box for a refined target proposal. Indicates the first The initial classification probability vector of a refined target proposal. Indicates the first The refined target proposal belongs to the category The probability, Indicates the first Confidence prediction score for a refined target proposal; For the collection of refined target proposals Each refinement target proposal, based on the bounding box For the largest scale feature map Perform the ROI Align operation to obtain the corresponding visual markers. ; S4: Will A visual marker Text features By concatenating the elements, we obtain extended text features. : ； S5: The final set of target proposals for detection is generated using the following method: S5.1: Extend text features and image features The pre-trained large model is input for secondary detection to obtain the bounding box of the target proposal. , , This indicates the number of target proposals obtained from the second detection; S5.2: Based on the bounding box of each target proposal From the maximum scale feature map Extract the corresponding feature sub-map ,in Represents bounding box The dimensions are then used to calculate the extended prediction score using the following formula. : ， Among them, superscript Indicates transpose. This represents the feature normalization function; S5.3: For refining the set of target proposals middle Each refined target proposal is based on its initial classification probability vector. Filter each category The maximum classification probability is denoted as the index of the refined target proposal corresponding to it. Then, the predicted score for each target proposal obtained from the secondary detection is obtained using the following formula. : ； S5.4: For each target proposal, based on the predicted score Calculate its classification probability vector and confidence level : ， ， in, This represents the softmax function; This yields the final set of target proposals. ; S6: Set of proposals for the final goal Post-processing is performed to obtain the final target detection result.

2. The adaptive target detection method during testing according to claim 1, characterized in that, The set of refining target proposals The generation method is as follows: S3.1: From the initial target proposal set Choose the one with the highest confidence level The initial target proposal is used as a salient target proposal. The value is determined based on actual needs; for each salient target proposal selected, the value is determined according to its initial classification probability vector. Determine the category , Indicates the first The original indexes of the salient target proposals are then used to construct the salient category set. ; S3.2: Initial target proposal set For each initial target proposal, based on the initial classification probability vector Determine the category ; Filtering and categorizing Belongs to the set of significant categories The initial target proposals constitute the preferred target proposal set. ; S3.3: Based on the set of salient categories Select the best target proposal set The target proposal set was obtained by categorizing them. , , Represents the set of salient categories Number of categories; for each target proposal set The non-maximum suppression algorithm is used for refinement to obtain the set of refined target proposals for the corresponding category. ;Will A collection of refined target proposals Merge to obtain a refined set of target proposals. .

3. The adaptive target detection method during testing according to claim 1, characterized in that, Step S5.4 also includes the final target proposal set. The optimization process involves setting a confidence threshold based on actual needs. If the confidence level of the target proposal If the target proposal is selected, it will be retained; otherwise, it will be deleted.

Images