An open-vocabulary object detection system and method using text-guided extrapolation and re-alignment
By employing text-guided extrapolation and realignment techniques, we provide new category visual supervision for open-vocabulary object detection, solving the problems of new category recognition and cross-domain robustness, and improving detection accuracy and recall. This technology is applicable to scenarios such as open-world perception, intelligent security, and autonomous driving.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TIANJIN UNIV
- Filing Date
- 2026-05-15
- Publication Date
- 2026-07-14
AI Technical Summary
Existing open-vocabulary object detection methods are insufficient in terms of new category recognition and cross-domain robustness. In particular, when faced with distribution shifts and style changes in input images, the models are prone to confusing new categories and misfiltering candidate boxes, resulting in decreased detection accuracy and recall, and high cost of retraining the detector.
By employing text-guided extrapolation and realignment, new category visual prototypes are generated using text semantics by freezing detector and pre-trained model parameters, constructing multi-domain enhanced visual embeddings, and mapping offset features back to source domain compatible distributions through a distribution-compatible realignment network. At the same time, candidate box confidence is adaptively corrected to improve the detection capability of new categories and domain-offset targets.
Without retraining the detector and pre-trained model, pseudo-visual supervision for new categories is provided, which improves the robustness and recall of the model in new categories and domain-shifted scenarios, reduces training and deployment costs, and is suitable for target detection in complex open environments.
Smart Images
Figure CN122391745A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence and computer vision technology, specifically involving open vocabulary object detection, visual language models, cross-modal representation learning and domain generalization techniques, and particularly an open vocabulary object detection method based on text-guided semantic extrapolation, distribution-compatible realignment, and adaptive correction of candidate box confidence. Background Technology
[0002] Open-vocabulary object detection aims to leverage the cross-modal semantic priors provided by pre-trained visual language models, enabling the detection model to identify novel category objects not seen during training, even when only basic category-labeled data is used during the training phase. Compared to traditional closed-set object detection, open-vocabulary object detection is no longer limited to a fixed set of categories, offering greater application flexibility and thus possessing significant value in scenarios such as open-world perception, intelligent security, autonomous driving, and general visual understanding.
[0003] Existing open-vocabulary object detection methods typically rely on text embeddings from visual language models as classifiers, calculating the similarity between visual features of candidate regions and text features of categories to achieve open-category recognition. However, these methods still have certain limitations in practical applications: the training phase usually only includes basic category annotations, lacking real visual supervision for new categories. When relying solely on text similarity for inference, the model is prone to confusion with new categories due to semantic similarity and missing visual prototypes, leading to insufficient accuracy in new category detection. Furthermore, input images in real-world deployment environments often exhibit significant style variations, imaging condition changes, and domain shifts, such as rainy, foggy, blurry, low-resolution, oil painting, watercolor, and cartoon images. These distribution shifts disrupt the alignment between original visual features and text semantics, reducing the reliability of cross-modal matching of candidate regions and significantly degrading the performance of open-vocabulary detection. In addition, the region proposal network in existing two-stage open-vocabulary detection methods is usually trained on source domain and basic category data, and its object scoring lacks adaptability to new categories or domain shift scenarios. Candidate boxes containing new category objects or style-variant objects are prone to ranking issues and failing to achieve top-K accuracy. During truncation or nonmaximum suppression, the system is misfiltered, resulting in a decrease in recall and further limiting the final detection performance. Meanwhile, some existing methods require retraining the entire detector or introducing additional complex modules to improve cross-domain robustness or new category recognition capabilities, leading to high training costs and high deployment complexity, which makes it difficult to meet the actual needs of low-cost enhancement of the capabilities of existing open vocabulary detection bases.
[0004] Therefore, how to construct effective visual supervision for new categories, alleviate representation drift caused by input distribution shift, and improve the recall capability of new categories and domain-shifted targets in the candidate box stage, without retraining the original detector and the main parameters of the pre-trained visual language model as much as possible, is a key technical problem in current open vocabulary object detection. Summary of the Invention
[0005] To address the problems of insufficient visual supervision for new categories, visual semantic alignment instability caused by cross-domain or cross-style inputs, and insufficient candidate box recall in existing technologies, this invention proposes an open-vocabulary object detection system and method based on text-guided extrapolation and realignment. This method, while freezing the main parameters of the original open-vocabulary detector and the main parameters of the pre-trained visual language model, utilizes the text semantic structure to generate visual prototypes of new categories and multi-domain enhanced visual embeddings for all categories. A lightweight distribution-compatible realignment network maps the offset features back to the source domain-compatible distribution. Simultaneously, adaptive correction is performed on the candidate box confidence during the inference stage, thereby improving the model's detection capability for new categories and domain-offset inputs.
[0006] To achieve the above objectives, the present invention adopts the following technical solution: An open-vocabulary object detection system with text-guided extrapolation and realignment, comprising an original region proposal network, a dataset, a text editor, an open-vocabulary detector, a text-guided semantic extrapolation module, a distribution-compatible realignment network, and a candidate box confidence adaptive correction module. The distribution-compatible realignment network consists of a visual realignment branch and a text realignment branch. The candidate box confidence adaptive correction module comprises an original region proposal network, an adaptive correction strategy unit, and a score fusion unit. The dataset is used to construct a training phase dataset and a test phase set of categories for the open vocabulary object detection task; the set of categories to be detected consists of an open vocabulary category set composed of a basic category set and a new category set. The text editor fills the category name into a preset text template to obtain the category text prompt, and encodes it into the basic category text embedding; The open vocabulary detector extracts the region features corresponding to the basic category bounding boxes and calculates the basic category visual prototype; The text-guided semantic extrapolation module uses the basic category text embedding center as a cross-modal anchor point to shift the category in the text space to the visual representation space to generate a new category visual prototype; at the same time, it uses the basic category visual prototype center as a cross-modal anchor point to shift the domain to the visual representation space to generate multi-domain enhanced visual embeddings for all categories. The distribution-compatible realignment network embeds multi-domain enhanced vision into the input vision realignment branch and the text realignment branch. Through category vision prototype supervision, text embedding supervision and discriminative constraints, it is trained to map the offset features back to the visual distribution and text semantic distribution that are compatible with the source domain. The adaptive correction of candidate box confidence retains the set of candidate boxes output by the original region proposal network and estimates the coarse-grained domain label of the current input based on the image content. Foreground semantic cue and background semantic cue are constructed based on the coarse-grained domain label, and the similarity between the candidate box and the foreground semantic cue is calculated to obtain the semantic foreground confidence. Then, the semantic foreground confidence is weighted and fused with the target confidence of the original region proposal network to obtain the corrected candidate box confidence.
[0007] Furthermore, the visual realignment branch employs a gated residual structure to map the enhanced visual embedding back to the source domain visual prototype of the corresponding category.
[0008] Furthermore, the text realignment branch employs a lightweight feedforward mapping structure to map the enhanced visual embeddings to a semantic space compatible with categorical text embeddings.
[0009] Furthermore, the process of mapping the offset features back to a visual distribution compatible with the source domain through category-based visual prototype-supervised training in the visual realignment branch includes: Based on arbitrary augmented samples Its visual realignment output is The cross-entropy loss is constructed as follows: in, Represents cosine similarity. Indicates the temperature coefficient; Visual realignment output is and categories Category Construct the reconstruction loss: Construct a contrastive loss based on the features of different samples in the visual realignment branch: in, For a set of sample indexes within a batch, Arbitrary anchor sample, A set of positive samples of the same category. Comparison of sample sets; , and These represent anchor samples, positive samples, and contrast samples, respectively. Construct the total loss for the visual realignment branch based on cross-entropy loss, reconstruction loss, and contrastive loss: in, , and This is the loss weighting coefficient.
[0010] Furthermore, the text realignment branch includes a process of mapping offset features back to a text semantic distribution compatible with the source domain through text embedding-supervised training, comprising: Based on the text embedding set of all categories Construct the text realignment branch loss: .
[0011] Furthermore, the confidence level of the corrected candidate box is: in, The weights are used to balance the contributions of the original candidate box target score and semantic prospect confidence. For semantic prospect confidence; The semantic prospect confidence is fused with the original region proposal network confidence.
[0012] This invention can also be implemented using the following technical solution: an open vocabulary target detection method based on text-guided extrapolation and realignment, comprising: Step 1: Construct training data and open category set: Use labeled image training data from the source domain as the basic training set. The training data only contains target labels from the basic category set. During the testing phase, the category set to be detected consists of an open vocabulary category set composed of the basic category set and the new category set. Step 2, extract basic category text embeddings and basic category visual prototypes: fill the category names into the preset text template to obtain the text embeddings of each category; extract the region-level visual features corresponding to the basic category annotation boxes from the frozen open vocabulary detector, and calculate the basic category visual prototypes; Step 3: Perform text-guided semantic extra-construction of new category visual prototypes: using the mean of the base category text embedding and the mean of the base category visual prototype as cross-modal anchors, transfer the offset of the new category text embedding relative to the center of the base category text to the visual representation space to generate new category visual prototypes. Step 4: Construct multi-domain enhanced visual embeddings for all categories: Construct text prompts containing domain descriptors for each category, where the domain descriptors are used to describe different styles or imaging conditions; Transfer the offset of the corresponding domain text embedding relative to the general category text embedding to the visual space and overlay it onto the visual prototype of each category to generate multi-domain enhanced visual embeddings, which are used to simulate the visual representation of candidate regions under domain offset or style offset. Step 5: The multi-domain enhanced visual embedding input distribution compatible realignment network is used to learn to map the offset features back to the visual distribution and text semantic distribution compatible with the source domain through category visual prototype supervision, text embedding supervision and discriminative constraints. Step 6, perform adaptive correction of candidate box confidence: During the inference phase, retain the candidate box set output by the original region proposal network and estimate the coarse-grained domain label of the current input based on the image content; construct foreground semantic cue and background semantic cue based on the domain label, calculate the similarity between the candidate box and the foreground semantic cue to obtain the semantic foreground confidence; then perform weighted fusion of the semantic foreground confidence with the target confidence of the original region proposal network to obtain the corrected candidate box confidence; Step 7, Output open vocabulary detection results: Input the corrected high-quality candidate boxes into the realignment network for feature remapping, and calculate the similarity between the candidate box features and the text embeddings of each category in the open vocabulary category set; fuse the classification scores of the visual realignment branch and the text realignment branch to obtain the target box detection results.
[0013] The construction of the high-quality candidate boxes includes: Based on the features of each candidate box Calculate temporary classification values using text embeddings from all categories: in, Indicates the temperature coefficient; The maximum temporary classification value is used as the confidence score for the candidate bounding box. 603. Set the filter threshold The set of high-confidence candidate boxes is obtained: in, This represents the set of high-confidence candidate boxes. Represents a single candidate box. Candidate boxes The confidence score; when the confidence score of the candidate box satisfies When that happens, it is included in the high-confidence candidate box set. ; 604. For the set of high-confidence candidate boxes Candidate boxes in the set undergo visual realignment and text realignment operations; candidate boxes that do not enter the high confidence set retain their original features and original classification scores.
[0014] Beneficial effects Compared with the prior art, the present invention has at least the following beneficial effects: 1. This invention constructs a visual prototype of a new category in the absence of real visual annotations for the new category through a text-guided semantic extrapolation mechanism, providing usable pseudo-visual supervision for new category recognition in open vocabulary detection and alleviating the semantic confusion caused by relying solely on text matching for new categories.
[0015] 2. This invention constructs a multi-domain enhanced visual embedding and introduces a distribution-compatible realignment network, enabling the model to learn to map cross-domain or cross-style shift features back to the source domain compatible distribution, thereby improving the robustness and stability of open vocabulary detection in domain-shift scenarios.
[0016] 3. This invention uses an adaptive correction strategy based on candidate box confidence to semantically enhance the target score of the original region proposal network, thereby reducing the probability of new category targets and domain-offset targets being falsely filtered in the candidate box stage and improving the candidate box recall capability.
[0017] 4. This invention only requires training a lightweight alignment module and does not require retraining the original detector body and the pre-trained visual language model body, which can enhance the capabilities of existing open vocabulary detection bases with lower training and deployment costs.
[0018] 5. This invention takes into account the generalization ability of new categories, cross-domain adaptability, and recall ability in the inference stage, and is suitable for target detection tasks in complex open environments, and has good engineering application value. Attached Figure Description
[0019] Figure 1 This is a schematic diagram of the overall process of the present invention.
[0020] Figure 2 A schematic diagram of a text-guided semantic extrapolation process and a distribution-compatible realignment network. Detailed Implementation
[0021] The following is in conjunction with the appendix Figure 1 - Appendix Figure 2 The technical solutions of this invention are clearly and completely described. All other embodiments obtained by those skilled in the art based on the technical solutions of this invention without inventive effort are within the scope of protection of this invention.
[0022] like Figure 1 , Figure 2As shown, this invention discloses an open vocabulary object detection system based on text-guided extrapolation and realignment. The system includes a dataset, a text editor, an open vocabulary detector, a text-guided semantic extrapolation module, a distribution-compatible realignment network, and a candidate box confidence adaptive correction module; wherein: Dataset Construction: This embodiment is designed for open-vocabulary object detection tasks. During the training phase, only basic category annotation data in the source domain can be accessed. During the testing phase, detection needs to be performed on an open-vocabulary category set that includes both basic and new categories. Furthermore, test images may exhibit style variations, image degradation, or cross-domain distribution shifts.
[0023] Let the set of basic categories be The new category set is Then the set of open vocabulary categories is: The training data only contains labeled targets from the base category set, while the testing phase requires additional data. The targets within are subjected to unified testing.
[0024] The text editor is used to extract the text embeddings of basic categories. The open vocabulary detector is used to extract basic category region-level visual prototypes. ,include: First, the category names are encoded using a text encoder from a pre-trained visual language model. For any category... Enter the category name into a preset text template, such as "a photo of a [category]", to obtain the category text hint, and encode it into the basic category text embedding. .
[0025] Secondly, region-level visual features corresponding to the base category bounding boxes are extracted from the frozen open vocabulary detector. For any base category... Let its regional feature set be The basic category region-level visual prototype can then be represented as: like Figure 2 As shown, the text-guided semantic extrapolation module uses the basic category text embedding center and the basic category visual prototype center as cross-modal anchor points to transfer the category offset and domain offset in the text space to the visual representation space, which is used to generate new category visual prototypes and multi-domain enhanced visual embeddings of all categories. Figure 2 The left side is a diagram illustrating the text-guided semantic extrapolation process.
[0026] This process uses the base category text embedding center and the base category region-level visual prototype center as cross-modal anchor points to transfer category and domain offsets from the text space to the visual representation space, generating new category visual prototypes and multi-domain enhanced visual embeddings for all categories. Specifically: Calculation Basic Category Text Center With the basic category regional visual prototype center As a reference anchor point for subsequent cross-modal extrapolation, the calculation formula is as follows: in, Represents the set of basic categories. Represents the basic category Text embedding, Represents the basic category The visual prototype.
[0027] To address the lack of visual supervision for new categories, this embodiment utilizes the differential transferability property of shared semantic space in visual language models to transfer category offsets from text space to visual space.
[0028] For any new category First, the text embedding is obtained from its category name. Then, based on the offset of the new category relative to the text center of the base category, a visual prototype of the new category is generated. : in, The scaling factor is used for semantic extrapolation of the new category, controlling the magnitude of text offset migration into the visual space. Essentially, this process projects the direction of category change in the text space onto the visual representation space to construct a new category visual prototype for subsequent training.
[0029] In addition to new category visual prototypes, visual shifts in different input domains are simulated using domain descriptors. Specifically, a pre-defined set of domain descriptors is used. This includes descriptions such as rainy days, foggy days, blurry, low resolution, oil painting, watercolor, and cartoon. (For any category...) and domain descriptors Constructing domain text hints: Input the domain text prompt into the frozen text encoder to obtain the corresponding domain text embedding. .
[0030] Further, define categories In the domain The text offset below is: Then, construct a multi-domain enhanced visual embedding based on the text offset: in, Indicates category The category prototype; when the category When it belongs to the basic category, When category When it belongs to a new category, . This is the domain offset scaling factor, used to control the intensity of the domain offset as it migrates into the visual space.
[0031] In this way, enhanced visual embeddings covering multiple domain offset directions can be uniformly generated for both the base category and the new category, providing training samples for subsequent distribution-compatible realignment networks.
[0032] The distribution-compatible realignment network comprises two parts: a visual realignment branch and a text realignment branch. The visual realignment branch maps the enhanced visual embeddings back to the vicinity of the source domain visual prototype corresponding to the category. The text realignment branch enhances the semantic consistency between region features and category text embeddings. Figure 2 As shown; the distribution-compatible realignment network is trained to map offset features back to a visual distribution and text semantic distribution compatible with the source domain through category visual prototype supervision, text embedding supervision, and discriminative constraints; the training process of the distribution-compatible realignment network is as follows: like Figure 2 As shown on the right, the distribution-compatible realignment network structure consists of two parts: the visual realignment branch DCR-V and the text realignment branch DCR-T. The visual realignment branch is used to map the enhanced visual embedding back to the vicinity of the source domain visual prototype of the corresponding category, while the text realignment branch is used to enhance the semantic consistency between region features and category text embeddings.
[0033] Since the enhanced visual embeddings generated by semantic extrapolation are not entirely equivalent to the region features in the real target domain, their distribution may differ from the feature distribution formed by the detector during source domain training. Therefore, this embodiment constructs a bi-branch distribution-compatible realignment network to perform regression correction on the enhanced visual embeddings.
[0034] The visual realignment branch, DCR-V, employs a gated residual structure. This enhances the visual embedding of the input. First, a feedforward network generates intermediate latent features and alignment residuals. Then, a gated branch controls the residual injection intensity. Finally, visual realignment features are output through residual superposition and normalization. Its form can be expressed as: in, This represents intermediate hidden features. Indicates alignment residual, Represents the gate vector, , and Indicates the parameters to be learned. This represents the activation function. This represents the Sigmoid function. This indicates element-wise multiplication.
[0035] The text realignment branch DCR-T employs a lightweight feedforward mapping structure, mapping the enhanced visual embeddings to a semantic space compatible with categorical text embeddings, which can be represented as follows: in, This indicates the output of the text realignment branch. This represents a lightweight feedforward mapping function.
[0036] During the training phase, the main parameters of the original detector and the pre-trained visual language model are frozen, and only the lightweight parameters of DCR-V and DCR-T are updated. Since there is no one-to-one correspondence between the enhanced embeddings generated by semantic extrapolation and the features of the real regions in the source domain, this embodiment uses category visual prototypes and category text embeddings as supervision signals.
[0037] First, regarding the basic categories Source region characteristics Perform aggregation to construct a visual prototype of the source domain category: Furthermore, construct an open vocabulary category set. Unified category prototype When category When it belongs to the basic category, let When category When it belongs to a new category, let The visual prototype of the new category is obtained by semantic extrapolation guided by the aforementioned text.
[0038] To simultaneously ensure distribution compatibility and class separability, the visual realignment branch DCR-V jointly optimizes the cross-entropy loss, reconstruction loss, and contrast loss.
[0039] Based on arbitrary augmented samples Its visual realignment output is The cross-entropy loss is constructed as follows: in, Represents cosine similarity. This represents the temperature coefficient.
[0040] Visual realignment output is and categories Category The reconstruction loss introduced by the construction is: Let the set of sample indices within a batch be... For any anchor sample The set of positive samples of the same category is denoted as The comparison sample set is denoted as Then we have: in, , and These represent the feature representations of anchor samples, positive samples, and contrast samples in the visual realignment branch, respectively.
[0041] Therefore, the total loss of the visual realignment branch is: in, , and This is the loss weighting coefficient.
[0042] For the text realignment branch DCR-T, cross-entropy loss based on class-specific text embeddings is used for supervision. Let the set of all class-specific text embeddings be... Then the text branch loss is: Finally, the overall loss of the distribution-compatible realignment network can be expressed as: Through the above training, the network can learn to realign feature perturbations caused by domain shift, style shift, or semantic extrapolation to the vicinity of the source domain compatible distribution, thereby improving consistency with the text semantic space while maintaining class separability.
[0043] The candidate box confidence adaptive correction module includes the original region proposal network (AdaRPN), an adaptive correction strategy unit, and a score fusion unit. The adaptive correction strategy unit performs four stages: domain estimation, semantic hints, calculation of semantic prospect confidence, and score fusion detection inference. Figure 1Includes an adaptive correction process for candidate box confidence: In open-vocabulary detection, even if the classifier has a certain ability to identify new categories, if candidate boxes are incorrectly filtered during the region proposal stage, the subsequent classification process cannot recover the target. Therefore, this embodiment introduces an adaptive correction strategy for candidate box confidence during the inference stage.
[0044] First, the candidate bounding box set, obtained from the original region proposal network output and after non-maximum suppression, is retained. Then, a subset of candidate regions is sampled from the candidate bounding box set, and their region-level visual features are extracted. Similarity calculation is performed between these regions and the text embeddings corresponding to a predefined set of domain descriptors to estimate the coarse-grained domain label corresponding to the current input image.
[0045] Based on the estimated domain labels, foreground and background semantic cues are constructed. For example, a foreground semantic cue can be represented as "this is an object in a [domain] image", and a background semantic cue can be represented as "this is a background area in a [domain] image". These are then encoded as foreground text embeddings and background text embeddings, respectively.
[0046] For any candidate bounding box region feature, calculate its similarity with the foreground text embedding and the background text embedding, respectively. The similarity score with the foreground cue is considered the semantic foreground confidence score, denoted as . Then, the semantic prospect confidence is compared with the original region proposal network confidence. The results are obtained by fusing the data to obtain the corrected candidate box confidence scores. in, The weights are used to balance the contributions of the original candidate box target score and semantic prospect confidence.
[0047] Based on the corrected candidate box confidence level Reorder the candidate boxes and retain the highest-scoring ones. Each candidate box is fed into the subsequent classification and regression branches. Through this method, without modifying the original region proposal network structure, semantic cues can be used to correct the target score of candidate boxes, thereby improving the recall rate of candidate boxes for new target categories and cross-domain targets.
[0048] In open-vocabulary detection, even if the classifier has a certain ability to identify new categories, if candidate boxes are incorrectly filtered during the region proposal stage, the subsequent classification process cannot recover the target. Therefore, this embodiment introduces an adaptive correction strategy for candidate box confidence during the inference stage.
[0049] Detection reasoning process During the inference phase, candidate boxes are reordered and filtered using an adaptive correction strategy based on their confidence levels. Subsequently, region-level visual features are extracted from the retained high-quality candidate boxes and fed into the trained DCR-V and DCR-T datasets to generate visual realignment features and text realignment features, respectively.
[0050] Given the large number of candidate boxes, to reduce additional inference overhead, we first analyze the features of each candidate box. Calculate the provisional classification score using text embeddings from all categories: in, Indicates the temperature coefficient; Furthermore, the maximum temporary classification probability is used as the confidence level of the candidate bounding box: Set the filter threshold The set of high-confidence candidate boxes is obtained: Only for the set of high-confidence candidate boxes Candidate boxes in the set undergo DCR-V and DCR-T realignment operations; candidate boxes that do not enter the high confidence set retain their original features and original classification scores.
[0051] Finally, the classification scores from the two branches are weighted and fused to obtain the final candidate box classification score: in, and These represent the DCR-V and DCR-T categories, respectively. The resulting classification scores, This represents the weight for the fusion of two branches.
[0052] Based on the final classification score The candidate bounding box prediction categories are determined, and the final object detection result is output by combining the bounding box regression results with non-maximum suppression post-processing. This inference method can improve the overall performance of open-vocabulary object detection in new categories and cross-domain scenarios while maintaining low overhead and taking into account visual distribution compatibility and text semantic consistency.
[0053] The system's open vocabulary target detection process includes: Step 1: Construct training data and open category set: Use labeled image training data from the source domain as the basic training set. The training data only contains target labels from the basic category set. During the testing phase, the category set to be detected consists of an open vocabulary category set composed of the basic category set and the new category set.
[0054] Step 2, extract basic category text embeddings and basic category region-level visual prototypes: input the category name into the frozen text encoder to obtain the basic category text embeddings; Extract the region-level visual features corresponding to the basic category bounding boxes from the frozen open vocabulary detector, and calculate the basic category region-level visual prototype; Step 3: Perform text-guided semantic extrapolation to generate a new category visual prototype: Using the mean of the base category text embeddings and the mean of the base category visual prototypes as cross-modal anchors, transfer the offset of the new category text embeddings relative to the center of the base category texts into the visual representation space to generate a new category visual prototype. This allows for the construction of visual supervision for new categories in the absence of real visual samples of the new categories.
[0055] in, This is a scaling factor for semantic extrapolation of the new category, used to control the magnitude of text offset migration into visual space.
[0056] Step 4: Construct multi-domain enhanced visual embeddings for all categories: Construct text cues containing domain descriptors for each category, whereby the domain descriptors describe different styles or imaging conditions; transfer the offset of the corresponding domain text embedding relative to the general category text embedding to the visual space and overlay it onto the visual prototype of each category to generate multi-domain enhanced visual embeddings, which are used to simulate the visual representation of candidate regions under domain or style offsets. .
[0057] in, Indicates category The category prototype; when the category When it belongs to the basic category, When category When it belongs to a new category, . This is the domain offset scaling factor, used to control the intensity of the domain offset as it migrates into the visual space.
[0058] Step 5: Train the distribution-compatible realignment network. Embed the multi-domain enhanced vision generated in Step 4 into the input vision realignment branch and the text realignment branch. Through category vision prototype supervision, text embedding supervision, and discriminative constraints, learn to map the offset features back to the visual distribution and text semantic distribution that are compatible with the source domain.
[0059] Step 6, perform adaptive correction of candidate box confidence: During the inference phase, retain the candidate box set output by the original region proposal network and estimate the coarse-grained domain label of the current input based on the image content; construct foreground semantic cue and background semantic cue based on the domain label, calculate the similarity between the candidate box and the foreground semantic cue to obtain the semantic foreground confidence; then perform weighted fusion of the semantic foreground confidence with the target confidence of the original region proposal network to obtain the corrected candidate box confidence.
[0060] Step 7: Output the open vocabulary detection results. Input the corrected high-quality candidate boxes into the realignment network for feature remapping, and calculate the similarity between the candidate box features and the text embeddings of each category in the open vocabulary category set. The classification scores of the visual realignment branch and the text realignment branch are fused to obtain the final category prediction results and the target box detection results.
[0061] Implementation Results Description This embodiment provides pseudo-visual supervision for new categories through a text-guided semantic extrapolation module without retraining the original detector body, mitigates representation drift under cross-domain input through a dual-branch distribution-compatible realignment module, and improves the recall rate of new categories and domain-shifted targets through an adaptive candidate box confidence correction strategy. Therefore, this method can simultaneously improve the overall performance of open-vocabulary object detection in both new category scenarios and distribution-shifted scenarios.
[0062] The above are merely preferred embodiments of the present invention. Those skilled in the art will recognize that various modifications and improvements can be made without departing from the principles of the present invention, and these modifications and improvements should also be considered within the scope of protection of the present invention.
Claims
1. An open lexical target detection system based on text-guided extrapolation and realignment, characterized in that, The system includes an original region proposal network, a dataset, a text editor, an open vocabulary detector, a text-guided semantic extrapolation module, a distribution-compatible realignment network, and a candidate box confidence adaptive correction module. The distribution-compatible realignment network consists of a visual realignment branch and a text realignment branch. The candidate box confidence adaptive correction module consists of an original region proposal network, an adaptive correction strategy unit, and a score fusion unit. middle: The dataset is used to construct a training phase dataset and a test phase set of categories for the open vocabulary object detection task; the set of categories to be detected consists of an open vocabulary category set composed of a basic category set and a new category set. The text editor fills the category name into a preset text template to obtain the category text prompt, and encodes it into the basic category text embedding; The open vocabulary detector extracts the region features corresponding to the basic category bounding boxes and calculates the basic category visual prototype; The text-guided semantic extrapolation module uses the basic category text embedding center as a cross-modal anchor point to shift the category in the text space to the visual representation space to generate a new category visual prototype; at the same time, it uses the basic category visual prototype center as a cross-modal anchor point to shift the domain to the visual representation space to generate multi-domain enhanced visual embeddings for all categories. The distribution-compatible realignment network embeds multi-domain enhanced vision into the input vision realignment branch and the text realignment branch. Through category vision prototype supervision, text embedding supervision and discriminative constraints, it is trained to map the offset features back to the visual distribution and text semantic distribution that are compatible with the source domain. The adaptive correction of candidate box confidence retains the set of candidate boxes output by the original region proposal network and estimates the coarse-grained domain label of the current input based on the image content. Foreground semantic cue and background semantic cue are constructed based on the coarse-grained domain label, and the similarity between the candidate box and the foreground semantic cue is calculated to obtain the semantic foreground confidence. Then, the semantic foreground confidence is weighted and fused with the target confidence of the original region proposal network to obtain the corrected candidate box confidence.
2. The system according to claim 1, characterized in that, The visual realignment branch uses a gated residual structure to map the enhanced visual embedding back to the source domain visual prototype of the corresponding category.
3. The system according to claim 1, characterized in that, The text realignment branch employs a lightweight feedforward mapping structure to map the enhanced visual embeddings to a semantic space compatible with categorical text embeddings.
4. The system according to claim 1, characterized in that, The visual realignment branch includes the process of mapping offset features back to a visual distribution compatible with the source domain through class-based visual prototype-supervised training, which includes: Based on arbitrary augmented samples Its visual realignment output is The cross-entropy loss is constructed as follows: in, Represents cosine similarity. Indicates the temperature coefficient; Visual realignment output is and categories Category Construct the reconstruction loss: Construct a contrastive loss based on the features of different samples in the visual realignment branch: in, For a set of sample indexes within a batch, Arbitrary anchor sample, A set of positive samples of the same category. Comparison of sample sets; , and These represent anchor samples, positive samples, and contrast samples, respectively. Construct the total loss for the visual realignment branch based on cross-entropy loss, reconstruction loss, and contrastive loss: in, , and This is the loss weighting coefficient.
5. The system according to claim 1, characterized in that, The text realignment branch includes the process of mapping offset features back to a text semantic distribution compatible with the source domain through text embedding-supervised training, which includes: Based on the text embedding set of all categories Construct the text realignment branch loss: 。 6. The system according to claim 1, characterized in that, The confidence level of the corrected candidate box is: in, The weights are used to balance the contributions of the original candidate box target score and semantic prospect confidence. For semantic prospect confidence; The semantic prospect confidence is fused with the original region proposal network confidence.
7. A text-guided extrapolation and realignment method for open-vocabulary target detection, characterized in that: The method implements open vocabulary target detection based on the system described in any one of claims 1-6, including: Step 1: Construct training data and open category set: Use labeled image training data from the source domain as the basic training set. The training data only contains target labels from the basic category set. During the testing phase, the category set to be detected consists of an open vocabulary category set composed of the basic category set and the new category set. Step 2, extract basic category text embeddings and basic category visual prototypes: fill the category names into the preset text template to obtain the text embeddings of each category; extract the region-level visual features corresponding to the basic category annotation boxes from the frozen open vocabulary detector, and calculate the basic category visual prototypes; Step 3: Perform text-guided semantic extra-construction of new category visual prototypes: using the mean of the base category text embedding and the mean of the base category visual prototype as cross-modal anchors, transfer the offset of the new category text embedding relative to the center of the base category text to the visual representation space to generate new category visual prototypes. Step 4: Construct multi-domain enhanced visual embeddings for all categories: Construct text prompts containing domain descriptors for each category, where the domain descriptors are used to describe different styles or imaging conditions; Transfer the offset of the corresponding domain text embedding relative to the general category text embedding to the visual space and overlay it onto the visual prototype of each category to generate multi-domain enhanced visual embeddings, which are used to simulate the visual representation of candidate regions under domain offset or style offset. Step 5: The multi-domain enhanced visual embedding input distribution compatible realignment network is used to learn to map the offset features back to the visual distribution and text semantic distribution compatible with the source domain through category visual prototype supervision, text embedding supervision and discriminative constraints. Step 6, perform adaptive correction of candidate box confidence: During the inference phase, retain the candidate box set output by the original region proposal network and estimate the coarse-grained domain label of the current input based on the image content; construct foreground semantic cue and background semantic cue based on the domain label, calculate the similarity between the candidate box and the foreground semantic cue to obtain the semantic foreground confidence; then perform weighted fusion of the semantic foreground confidence with the target confidence of the original region proposal network to obtain the corrected candidate box confidence; Step 7, Output open vocabulary detection results: Input the corrected high-quality candidate boxes into the realignment network for feature remapping, and calculate the similarity between the candidate box features and the text embeddings of each category in the open vocabulary category set; fuse the classification scores of the visual realignment branch and the text realignment branch to obtain the target box detection results.
8. The method according to claim 7, characterized in that: The construction of the high-quality candidate boxes includes: Based on the features of each candidate box Calculate temporary classification values using text embeddings from all categories: in, Indicates the temperature coefficient; The maximum temporary classification value is used as the confidence score for the candidate bounding box. Set the filter threshold The set of high-confidence candidate boxes is obtained: in, This represents the set of high-confidence candidate boxes. Represents a single candidate box. Candidate boxes The confidence score; when the confidence score of the candidate box satisfies When that happens, it is included in the high-confidence candidate box set. ; For high confidence candidate box set Candidate boxes in the set undergo visual realignment and text realignment operations; candidate boxes that do not enter the high confidence set retain their original features and original classification scores.