Task-oriented visual language target detection method and system

Abstract

The application discloses a task-independent incremental visual language target detection method and system, and relates to the technical field of target detection.The method comprises the following steps: acquiring continuously-arriving target detection task data; constructing a pre-training open-vocabulary target detection model based on an attention mechanism and a class-level low-rank adaptive expert library; training the pre-training open-vocabulary target detection model by using the target detection task data, wherein an enhanced training prompt is constructed by random prompting enhancement; performing task-independent reasoning on a test image by using the trained pre-training open-vocabulary target detection model, dynamically weighting and fusing class-level experts in the class-level low-rank adaptive expert library according to the Mahalanobis distance between the features and the class distribution of the test image, and obtaining a target bounding box, a class name and a confidence degree.The application enables the model to complete target detection in the face of a unified prompt composed of all learned classes without the need for task identity priori.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a visual language target detection method and system oriented towards task-independent incremental approaches. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Open vocabulary object detection models based on visual language pre-training suffer from significant performance degradation in object detection tasks because the image distribution, object scale, background complexity, and category combinations in real-world scenes often differ from those in large-scale pre-training data.

[0004] Some studies have attempted to address this problem through incremental visual language object detection. The model learns new categories or domains sequentially according to the task flow, while avoiding forgetting existing skills and maintaining the ability to generalize to unseen categories.

[0005] However, existing incremental visual language object detection methods mostly employ task-incremental evaluation protocols. These protocols provide task identity during testing, and the model only needs to perform detection within the category prompts for the current task. This setup facilitates the analysis of stability and adaptability, but it falls short of real-world deployment. Task-agnostic incremental visual language object detection, on the other hand, does not provide any task priors during the inference phase; the model must simultaneously consider all learned categories. However, cross-task categories in long prompts can cause semantic interference, making it difficult for visual features to consistently focus on the correct text terms. Furthermore, category experts who learn independently in different tasks may differ significantly in parameter orientation, and direct averaging and fusion can lead to parameter conflicts. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the present invention aims to provide a visual language target detection method and system oriented towards task-agnostic incremental approaches. It proposes a task-agnostic distribution-aware adaptation (TADA) method, which enhances the robustness of the model in task-agnostic scenarios by combining random prompts during the training phase with distribution-aware fusion during the testing phase. This enables the model to complete target detection in the face of unified prompts composed of all learned categories without prior knowledge of task identity.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solution: The first aspect of this invention provides a visual language target detection method oriented towards task-independent increments, comprising the following steps: Acquire continuously arriving target detection task data, including training images, the set of bounding boxes corresponding to the training images, and the category labels corresponding to the bounding boxes, and preprocess the training images; Construct a pre-trained open-vocabulary target detection model based on attention mechanism and a category-level low-rank fitting expert database; The pre-trained open vocabulary object detection model is trained using object detection task data. In this process, enhanced training prompts containing the current category, historical categories, and random noise words are constructed through random prompts. The pre-trained open vocabulary target detection model is used to perform task-independent inference on the test image. Based on the Mahalanobis distance between the test image features and the category distribution, the category-level experts in the category-level low-rank adaptation expert library are dynamically weighted and fused to obtain the target bounding box, category name and confidence score.

[0008] Furthermore, preprocessing operations such as normalization, pixel normalization, and batch processing are performed on the training images.

[0009] Furthermore, the attention-based pre-trained open vocabulary object detection model includes an image encoder, a text encoder, a cross-modal feature fusion module, a detection decoder, and a detection head.

[0010] Furthermore, during training, a matching algorithm is used to match the predicted bounding box with the ground truth bounding box. Only the ground truth labeled categories of the training image are used for matching and loss calculation, and the mean of category features and the shared covariance matrix are maintained online.

[0011] Furthermore, during the task-independent reasoning process of the pre-trained open vocabulary target detection model on the test image, the task identity of the test image is not obtained. All learned categories are used to form a unified reasoning prompt. The Mahalanobis distance between the test image features and the mean of each category is calculated. The fusion weight of the category-level low-rank fitting experts is obtained through a normalized exponential function with a temperature coefficient. Then, the expert parameters are dynamically fused and the target bounding box, category name, and confidence score are output.

[0012] Furthermore, during the training process, in the parameter update phase, if the training samples contain the true category, the low-rank expert and shared projection matrix corresponding to that true category are updated; the basic weights, the main pre-trained weights of the image encoder, and the main pre-trained weights of the text encoder remain frozen or are frozen according to a preset strategy.

[0013] A second aspect of the present invention provides a visual language target detection system oriented towards task-independent incremental methods, comprising: The data acquisition and preprocessing module is configured to acquire continuously arriving target detection task data, including training images, a set of bounding boxes corresponding to the training images, and category labels corresponding to the bounding boxes, and to preprocess the training images. The model and expert database construction module is configured to build a pre-trained open vocabulary object detection model based on an attention mechanism and a category-level low-rank fitting expert database. The model training module is configured to train a pre-trained open vocabulary object detection model using object detection task data, wherein enhanced training prompts containing the current category, historical categories and random noise words are constructed through random prompts. The task-independent reasoning module is configured to perform task-independent reasoning on test images using a pre-trained open-vocabulary object detection model. It dynamically and weightedly fuses category-level experts from a category-level low-rank adaptation expert library based on the Mahalanobis distance between test image features and category distribution to obtain the target bounding box, category name, and confidence score.

[0014] A third aspect of the present invention provides a computer-readable storage medium storing a computer program adapted to be loaded by a processor and to execute steps in the task-independent incremental visual language target detection method as described in the first aspect of the present invention.

[0015] A fourth aspect of the present invention provides a computer device comprising: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the task-independent incremental visual language target detection method as described in the first aspect of the present invention.

[0016] A fifth aspect of the present invention provides a computer program product or computer program comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the task-independent incremental visual language target detection method as described in the first aspect of the present invention.

[0017] The above one or more technical solutions have the following beneficial effects: This invention discloses a visual language target detection method and system for task-agnostic incremental approaches, namely Task-Agnostic Distribution-Aware Adaptation (TADA). This method acquires continuously arriving target detection tasks and constructs a category-level low-rank adaptation expert database based on a pre-trained open-vocabulary target detection model. During the training phase, it utilizes the current category, historical categories, and random noise words to construct enhanced training prompts, enabling the model to fuse image and text features in complex cross-task text environments, and calculates the detection loss based solely on the ground truth labeled categories. Simultaneously, it maintains the mean values ​​of each category's features and the shared covariance matrix online based on image encoding features. During the inference phase, the task identity of the test image is not acquired; all learned categories form a unified inference prompt. The Mahalanobis distance between the test image features and the mean values ​​of each category is calculated, and the fusion weights of the category-level low-rank adaptation experts are obtained through a temperature-coefficient normalized exponential function. The expert parameters are then dynamically fused, and the target bounding box, category name, and confidence score are output. This invention can alleviate semantic interference caused by unified prompts and cross-task expert parameter conflicts, improving target detection accuracy in task-agnostic open-vocabulary scenarios.

[0018] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of the visual language target detection method for task-independent increments in Embodiment 1 of the present invention; Figure 2 Comparison chart of detection results under standard incremental settings and task-independent settings; Figure 3 A visual illustration of the attention of image features to the correct target category under different cue conditions. Detailed Implementation

[0021] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0022] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof. The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0023] Object detection is a fundamental task in computer vision. Traditional detectors typically rely on a fixed set of categories for training and inference. As application scenarios change, fixed-category detectors require re-collecting data and retraining, making it difficult to meet the demands of continuously expanding open-world applications. In recent years, open-vocabulary object detection models based on visual language pre-training have emerged. These models can utilize text queries to describe categories, enabling them to identify objects outside the training categories, thus providing a new technical approach for open-scene detection.

[0024] However, open-vocabulary detection models still experience significant performance degradation when facing specialized downstream domains. This is because the image distribution, target scale, background complexity, and category combinations in real-world scenes often differ from those in large-scale pre-training data. Incremental visual language object detection attempts to address this issue: the model learns new categories or new domains sequentially according to the task flow, while avoiding forgetting existing capabilities and maintaining generalization ability for unseen categories.

[0025] Existing incremental visual language object detection methods mostly employ task-incremental evaluation protocols. These protocols provide task identity during testing, allowing the model to perform detection only within the category cues provided for the current task. This setup facilitates the analysis of stability and adaptability, but it falls short of real-world deployment. In open environments, the system typically cannot know in advance which task the input image belongs to, nor can it provide the model with only a few in-task categories. The model needs to contend with long cues comprised of all historical task categories and complete recognition within a unified semantic space.

[0026] Some studies focus on task-agnostic incremental visual-language object detection. In this setting, no task priors are provided during the inference phase, and the model must consider all learned categories simultaneously. Specifically, suppose the continuous learning process includes several tasks, each corresponding to a set of categories. Standard incremental visual-language object detection uses only the categories from that task to construct text cues when testing a particular task. The task-agnostic setting removes this restriction; during the testing phase, the union of all learned categories is used as the cue, and the model needs to simultaneously locate and classify candidate categories across tasks in a single inference. Regarding evaluation methods, standard protocols typically evaluate the test set for each task separately; task-agnostic protocols, on the other hand, merge test samples from different tasks into a global test set. For example... Figure 2 As shown, the standard incremental setting only completes recognition within a single-task prompt, while the task-independent setting requires judgment under a unified prompt encompassing multiple task categories, thus more easily exposing cross-task false positives. This significantly increases the number of negative samples faced by each category, especially amplifying the risk of false positives in low-sample tasks. Therefore, this protocol can more directly measure the robustness of the model in real-world open environments.

[0027] Visual language detectors typically rely on deep fusion between image features and text cues for recognition. If only in-task categories are used during training, the model learns relatively sparse, single-task cross-modal alignment relationships; however, if all task categories are suddenly input during inference, the cue length and semantic diversity increase significantly, and the attention to correct category words is easily diluted by irrelevant categories. For example... Figure 3 As shown, single-task prompts allow the model to focus more intently on the target region, while cross-task prompts significantly weaken the response to the correct target category. This cueing interference reduces the correspondence between the target region and the correct text category, leading to an increase in both false negatives and false positives.

[0028] To avoid catastrophic forgetting, modular incremental methods often maintain independent low-rank adaptation (LoRA) experts for different categories and preserve a common representation by sharing parameters. This strategy is more effective for intra-task inference because category experts within the same task usually have similar optimization trajectories; however, in task-agnostic inference, experts from different tasks need to be fused together. If all expert parameters are simply averaged, modules with large differences in direction will cancel each other out or destroy the learned category adaptation capabilities, resulting in obvious cross-task parameter conflicts.

[0029] In summary, current task-agnostic incremental visual language object detection faces two key challenges: First, cross-task categories in long prompts can cause semantic interference, making it difficult for visual features to consistently focus on the correct text terms; second, category experts who learn independently in different tasks may have significant differences in parameter orientation, and direct averaging and fusion can lead to parameter conflicts.

[0030] To address the aforementioned challenges, this embodiment proposes the TADA framework, which enhances the model's robustness in task-agnostic scenarios by fusing random cue enhancement during the training phase with distribution-awareness during the testing phase. Details are provided in the following embodiment.

[0031] Example 1: Embodiment 1 of the present invention provides a visual language target detection method oriented towards task-independent incremental methods, such as... Figure 1 As shown, it includes the following steps: S1: Obtain continuously arriving target detection task data, including training images, the set of bounding boxes corresponding to the training images, and the category labels corresponding to the bounding boxes, and preprocess the training images.

[0032] S1.1: Obtain data from the continuous arrival of target detection tasks.

[0033] In one specific implementation, the continuous learning process of this embodiment includes N sequentially arriving target detection tasks. , ... The data for the k-th task is represented as follows: ,in For the input training image, This is the set of bounding boxes in the image. It can be represented as , The category label for the bounding box. and These represent the x and y coordinates of the top-left vertex of the bounding box, respectively. and These represent the x and y coordinates of the bottom right corner vertex of the bounding box, respectively.

[0034] set up This is the set of category names. The set of all learned categories is... Where i represents the index of the sample and j represents the image. Index of the target bounding box. Representing an image The true category label of the j-th bounding box. Let represent the set of category names for the i-th incremental task; therefore, the set of categories for the k-th task is . The first k tasks have all learned categories as .

[0035] S1.2: Preprocessing operations for training images, including normalization, pixel normalization, and batch processing.

[0036] In one specific implementation, during the data input phase, the training images are... Size normalization, pixel normalization, and batch processing are performed to obtain the model input tensor. Where R represents the set of real numbers, B is the batch size, and H and W are the height and width of the input image. Text cues are constructed for each category name c, such as "a photo of c" or "c", and multiple category names are concatenated into a cue sequence P.

[0037] S2: Construct a pre-trained open vocabulary object detection model based on attention mechanism and a category-level low-rank fitting expert database.

[0038] S2.1: Construct a pre-trained open vocabulary target detection model based on attention mechanism.

[0039] In one specific implementation, the pre-trained open vocabulary object detection model based on the attention mechanism can use Grounding DINO as the basic open vocabulary detector. DINO in Grounding DINO is a detection Transformer model based on improved denoised anchor boxes, and Transformer is a neural network structure based on the self-attention mechanism.

[0040] Specifically, attention-based pre-trained open-vocabulary object detection models include image encoders. Text encoder Cross-modal feature fusion module Detection decoder and detection head .

[0041] Among them, the image encoder outputs visual features. ,in It can be multi-scale features or a flattened visual token sequence; the text encoder takes the cue sequence P as input and outputs text features. The cross-modal feature fusion module supports... and Perform self-attention and cross-attention calculations to obtain fused features. .

[0042] It is important to note that although the text encoder takes cue sequences as input, during the training phase, the input is augmented with random cues. During the inference phase, the input is a unified hint containing all learned categories. Before actual input, the category names can be concatenated according to preset delimiters and segmented to obtain a text token sequence.

[0043] The attention calculation in this embodiment is: Where Q, K, and V are the query, key, and value obtained by linear mapping of visual or text features, respectively, and d is the feature dimension. Through this calculation, the visual token can adjust the response based on the text category token, and the text category token can also participate in the semantic alignment of the target region.

[0044] LoRA experts can insert image-text fusion layers, query / key / value projection layers in attention layers, or linear layers in feedforward networks.

[0045] Specifically, in the linear transformation layer of the pre-trained open vocabulary detector, the original weights are updated directly instead of directly using category-related low-rank increments. In a specific embodiment of Grounding DINO, the target linear layer is located within the Transformer cross-modal feature fusion module, either in the query, key-value projection layer of the attention layer, or a linear layer in the feedforward network. This module receives visual / text tokens output from the image encoder and text encoder, and performs image-text fusion using self-attention and cross-attention. LoRA experts are used to perform category-related adaptation on these fusion transformations.

[0046] S2.2: Construct a category-level low-rank fitting expert database based on a pre-trained open vocabulary object detection model.

[0047] In one specific implementation, when constructing a category-level low-rank fitting expert database, the base weights of the pre-trained open vocabulary object detection model are... Keep frozen. For the target linear layer in the model, set the adaptation weights for class c to... ,in Let c be a low-rank expert matrix. To share the projection matrix, r is the low-rank dimension. and Let be the input and output dimensions of the target linear layer, respectively. For the input feature h, the output of this layer can be written as... Therefore, each category only needs to store a smaller number of... The shared projection matrix B is used to maintain a universal representation across categories.

[0048] In this embodiment, the target linear layer is the linear transformation layer to be adapted in the cross-modal feature fusion module, specifically the query / key / value projection layer and feedforward network linear layer in the Transformer attention module. Its input... It is the visual token, text token, or fusion token output from the previous fusion sublayer that enters the linear layer, originating from the pre-processing of the image encoder, text encoder, or cross-modal fusion module; its original output is... After adding category experts, the output is: During inference, W(x) is constructed based on the fusion weights obtained from the input image, and then W(x) performs the corresponding linear transformation. The subsequent results are fed into the remaining layers of cross-modal fusion, the detection decoder, and the detection head.

[0049] S3: Train the pre-trained open vocabulary object detection model using object detection task data.

[0050] In one specific implementation, augmented training cues are constructed by using random prompts, which include the current category, historical categories, and random noise words. Specifically, during training for the k-th task, the current task category set is first used as the basis for the augmented training cues. Collection of historical learned categories and random noise word set Constructing Enhanced Training Tips ,in from Randomly selected from among them. It can be composed of random letters, random subwords, random numbers, or semantically neutral placeholder strings, and is used to simulate future queries of unknown categories or irrelevant open vocabulary.

[0051] In this embodiment, the role of enhanced training cues is to artificially construct complex textual contexts that more closely resemble task-agnostic reasoning during training. These contexts consist of the current task category, categories randomly drawn from historical tasks, and random noise words. Historical categories enable the model to learn to distinguish the current target from previously learned concepts within the same cues; random letters, random subwords, random numbers, or semantically neutral placeholder strings are used to simulate unknown interference in future task categories and open-word queries, and the amount of noise is randomly varied to simulate unified cues of different lengths. This reduces the distributional differences between single-task cues during training and cross-task unified cues during inference, minimizing attention dilution and semantic interference.

[0052] For training images Image encoder output Text encoder output The cross-modal feature fusion module outputs The decoder outputs the target query features. The detection head outputs a prediction set. ,in For the m-th prediction box, It corresponds to the category score vector and confidence level.

[0053] During the loss calculation phase, a matching algorithm is used during training to match predicted bounding boxes with ground truth bounding boxes. Only the ground truth labeled categories from the training images are used for matching and loss calculation, and the mean of category features and the shared covariance matrix are maintained online. Specifically, maintaining the mean of category features... The shared covariance matrix Σ is used to represent the central position of category c in the image feature space, while it describes the scale differences and correlations between different feature dimensions. Together, they constitute a statistical description of the category distribution, allowing Mahalanobis distance to be used during inference to determine the proximity of the input image to the visual distribution of category c, thereby determining the fusion weights of the category experts. These values ​​are updated online with samples during the training phase: for new image features belonging to category c, the category count is updated first. and mean Then, the shared scatter matrix S is calculated using the old and new means, and finally... The shared covariance matrix is ​​obtained.

[0054] The total loss in this embodiment can be expressed as: .

[0055] in, For the total loss, For category classification loss, For bounding box L1 regression loss, For generalized intersection and comparison of losses, , These are the weighting coefficients.

[0056] Historical categories in this embodiment and random noise words This implementation limits historical categories and random noise terms to only participate in cross-modal fusion, excluding them from positive / negative sample matching and loss calculation for the true category. This ensures that loss supervision corresponds only to the true annotations in the training images, preventing semantically neutral noise from being mistakenly treated as a positive class. It also avoids generating additional erroneous negative sample constraints from historical categories when there is no target in the current image, reducing the destruction of learned knowledge. Furthermore, this calculation still allows the fusion module to encounter complex cross-task textual contexts during training, thereby improving robustness to unified cross-task prompts.

[0057] During training, in the parameter update phase, if the training samples contain the true class c, then the low-rank expert corresponding to that true class c is updated. And shared projection matrix B; basic weights The main pre-trained weights of the image encoder and text encoder are kept frozen or frozen according to a preset strategy. This reduces the disruption to the pre-trained open vocabulary knowledge when learning new categories. Specifically, the main pre-trained weights of the image encoder are parameters of the visual branch; in a specific implementation, the Swin-Tiny image backbone of the Grounding DINO model and related parameters extracted from its visual features are used. The main pre-trained weights of the text encoder are parameters of the language branch, used to map category name cue sequences to text features. In the open vocabulary pre-training of Grounding DINO, the image branch, text branch, and cross-modal fusion parameters jointly learn the image-text alignment capability.

[0058] In the category distribution statistics phase, the image encoder uses training images Output global image features This is used to update the class mean and covariance, representing the current training sample using global image features or pooled visual features. Class feature mean is the average vector of image features corresponding to training samples of category c, used to represent the distribution center of that category; the shared scatter matrix S is the cumulative matrix of intra-class feature biases for all learned categories, used to estimate the common variance and inter-dimensional correlations of each feature dimension. For category c, maintain sample counts. , categorical feature mean And share the scatter matrix S. When new features When it belongs to category c, first save the mean of the old category features. Then let , and according to Update the mean, by Update the scatter matrix. The shared covariance matrix is... ,in The diagonal regularization term is used to avoid singularity in the covariance matrix and ensure stability in inversion. The symbol represents the assignment / online update symbol, indicating that the old value of the variable on the left is replaced with the new value calculated on the right. and Used together The smaller the distance, the closer the test image features are to the visual distribution of category c, and the larger the corresponding category LoRA expert weight, where x is the test image and v is the test image feature.

[0059] The shared covariance matrix Σ is obtained by normalizing the shared scatter matrix S and adding a regularization term. It is used to characterize the magnitude of variance and the correlation between dimensions of all learned class features in the common visual feature space: directions with larger variances are appropriately weighted, and correlated feature dimensions are not incorrectly treated as independent. During inference, it is utilized... Calculating Mahalanobis distance allows for a more accurate measurement of test image features and class mean than Euclidean distance. The distribution distance is used to provide a basis for category-level LoRA expert fusion.

[0060] S4: Use the pre-trained open vocabulary object detection model to perform task-independent reasoning on the test image. Based on the Mahalanobis distance between the test image features and the category distribution, dynamically weight and fuse category-level experts from the category-level low-rank adaptation expert library to obtain the target bounding box, category name, and confidence score.

[0061] In one specific implementation, this embodiment utilizes a pre-trained open vocabulary target detection model to perform task-independent reasoning on test images. During this process, the task identity of the test image is not obtained. All learned categories are used to form a unified reasoning prompt. The Mahalanobis distance between the test image features and the mean of each category is calculated. The fusion weight of the category-level low-rank fitting experts is obtained through a normalized exponential function with a temperature coefficient. Then, the expert parameters are dynamically fused and the target bounding box, category name, and confidence score are output.

[0062] Specifically, in the task-independent reasoning phase, for the test image x, the task identity is not received; instead, all learned categories are considered. Constituting a unified reasoning hint The image encoder obtains the features of the test image based on the test image. The text encoder obtains uniform prompt features. For each learned category c, calculate the Mahalanobis distance. This distance takes into account the correlation and scale differences between feature dimensions, making it more suitable for class distribution matching in high-dimensional visual feature spaces than the Euclidean distance.

[0063] The Mahalanobis distance is converted into category expert fusion weights, calculated as follows: ,in This represents the temperature coefficient. The smaller the distance, the greater the weight, indicating that the test image is closer to the visual distribution corresponding to that category of expert. The Mahalanobis distance between the test image and category j.

[0064] Based on the dynamic combination of category-level LoRA experts using fusion weights, the adaptation weights relevant to the input image are obtained. Then use It participates in the calculation of the target linear layer and completes cross-modal fusion, detection decoding, and detection head prediction. It is a target linear layer adaptation weight dynamically generated for the current input image. For the input feature h entering the target linear layer, the actual output is... The output is then fed into the cross-modal fusion module for further computation, the detection decoder, and the detection head, ultimately yielding the target location and category prediction. Because experts whose distribution is closer to the current image have higher weights, interference from irrelevant experts is suppressed. Compared to static average experts, dynamic fusion reduces the interference of irrelevant category experts on the current image prediction.

[0065] After the detection head outputs multiple candidate boxes and class scores, it first filters low-confidence candidate boxes based on the confidence threshold, then performs non-maximum suppression to remove highly overlapping redundant boxes, and finally obtains the target bounding box coordinates, class name, and confidence score.

[0066] The purpose of this invention is to provide a method for fusion of cue enhancement and distribution awareness in task-agnostic incremental visual language object detection, enabling the model to complete object detection with unified cuees composed of all learned categories without prior task identity knowledge. To achieve this objective, the invention employs the following technical solution: acquiring continuously arriving object detection task data; constructing a category-level low-rank fitting expert library on a pre-trained open-vocabulary object detection model; constructing enhanced training cuees containing the current category, historical categories, and random noise words during the training phase; calculating the detection loss only for the ground truth labeled categories during training, and maintaining the category feature mean and shared covariance matrix online; and dynamically weighting and fusing category-level experts based on the Mahalanobis distance between the test image features and the category distribution during the inference phase, outputting the target bounding box, category name, and confidence score.

[0067] This invention can reduce the distribution gap between training prompts and task-independent inference prompts through random prompts. It can also reduce the disruption of pre-trained knowledge by new task learning by freezing base weights and training category-level low-rank experts. Finally, this invention uses Mahalanobis distance and a shared covariance matrix for distribution-aware fusion to suppress expert contributions irrelevant to the current image. This invention can improve detection accuracy and maintain zero-shot generalization capability in open incremental scenarios where task identity is unknown.

[0068] To better illustrate the superiority of the method of this invention, the following experiments were conducted. In the experiments, the ODiW-13 continuous object detection task was used as the downstream task flow, and the publicly available general visual object detection dataset MSCOCO (Microsoft Common Objects in Context) was used to evaluate zero-shot capability. The evaluation metric was mean average precision (mAP). Experimental results show that random prompt enhancement and test-time distributed awareness fusion can alleviate semantic interference and parameter conflicts under unified cross-task prompts.

[0069] The experiments were primarily conducted on the OdinW-13 continuous detection task. This benchmark includes several subsets with significant domain differences, which can be used to evaluate the model's ability to continuously adapt to real-world complex scenarios. The model sequentially learns 13 downstream tasks and is uniformly evaluated after all training is complete. To test zero-shot capability, the experiments also report zero-shot mean accuracy on MS COCO.

[0070] This experiment employs two evaluation protocols. The first is a task-independent protocol, where all samples share a unified cue encompassing all learned categories during the testing phase, and the average precision is calculated on the merged test set. The second is a standard incremental protocol, where task identities are used to construct in-task cuees during the testing phase to observe whether the method remains competitive under more lenient settings. Evaluation metrics include downstream task average mAP and COCO zero-sample mAP.

[0071] To facilitate understanding of the subsequent tables, a brief description of the baseline methods used in the comparison is provided first. Grounding DINO, as a pre-trained open vocabulary detector, is used to reflect the zero-shot detection capability without downstream incremental adaptation. TFA and iDETR represent traditional incremental object detection approaches, mainly focusing on continuous learning between new and old categories. ZiRa and DitHub are representative methods for incremental visual language object detection; the former emphasizes maintaining zero-shot generalization capability when learning new tasks, while the latter uses a modular LoRA expert database for category expansion. The comparison of our proposed method TADA with these baselines is mainly used to verify its comprehensive advantages in standard incremental settings, task-independent unified cue settings, and zero-shot generalization preservation. Tables 1 and 2 show the main results of ODinW-13 under task-independent settings and standard incremental settings, respectively.

[0072] Table 1. Main results of ODinW-13 under task-independent settings

[0073] Table 2. Main results of ODinW-13 under standard incremental settings

[0074] As shown in Table 1, under the task-independent setting, TADA's average mAP for OdinW-13 reached 64.5, significantly higher than the modular baseline DitHub's 32.5. This result indicates that simply relying on category expert isolation cannot resolve cross-task conflicts in the open reasoning stage; only by enhancing the robustness of cues during training and dynamically selecting experts according to the input distribution during inference can the complex interference brought about by unified cues be effectively addressed.

[0075] As shown in Table 2, under the standard incremental settings, TADA's average mAP is 67.5, which is a slight improvement compared to DitHub. This indicates that distributed-aware fusion is not only effective for task-independent protocols; even in scenarios with more lenient task identities, dynamic weights can further reduce fusion errors among category experts.

[0076] From the zero-shot performance perspective, TADA maintains an mAP of 47.2 on COCO, which is close to the level of 48.4 of the pre-trained model and outperforms most incremental learning baselines. This indicates that the method does not severely compromise the ability of the original open vocabulary when learning downstream tasks, and can effectively balance downstream adaptation and basic generalization. Table 3 shows the relevant results of the few-shot experiments.

[0077] Table 3 Results of Few-Shot Continuous Learning

[0078] As shown in Table 3, the few-shot experiments further validated the stability of the method. Under the 1-shot, 5-shot, and 10-shot configurations, TADA achieved the highest or near-highest average accuracy in both the standard incremental and task-independent incremental settings. Particularly in the task-independent setting, TADA's advantage continued to expand as the number of samples increased from 1-shot to 10-shot, indicating that random cue enhancement and dynamic expert routing can more fully utilize limited labeled data.

[0079] The results of the relevant ablation experiments and the comparison of the enhancement strategies are shown in Tables 4 and 5.

[0080] Table 4 Ablation Results of Core Modules

[0081] Table 5 Comparison of different prompt enhancement strategies

[0082] As shown in Table 4, the ablation experiments demonstrate that adding distribution-aware feedback alone during testing improved the task-independent average mAP from 32.5 to 45.9, indicating that parameter conflict is a significant cause of performance degradation. Further adding random prompts increased the average mAP to 64.5, demonstrating that actively exposing the model to complex prompting environments during training is crucial for final task-independent inference.

[0083] As shown in Table 5, the comparison of suggestion enhancement strategies reveals that random noise terms are more effective than external true categories in maintaining zero-shot capability. This is because while external true categories can improve robustness as perturbations, these categories themselves have explicit semantics. If treated as negative perturbations over a long period, they may weaken the model's ability to identify true categories. Random noise terms, lacking specific semantics, are more suitable as neutral perturbations.

[0084] Regarding distance metrics, the task-independent average mAP using Mahalanobis distance is 64.5, significantly higher than the 51.3 of Euclidean distance. This result demonstrates that considering covariance structure and dimensionality correlation in high-dimensional visual feature spaces helps to more accurately assess the matching relationship between images and category experts.

[0085] This invention focuses on task-agnostic incremental visual language object detection, pointing out that existing methods rely excessively on task identity priors during the testing phase, making it difficult to meet the needs of real-world open scenarios where task boundaries are unknown. Under unified cross-task prompts, the model encounters two core problems: prompt interference and parameter conflicts, leading to significant performance degradation of existing incremental adaptation strategies.

[0086] To address the aforementioned issues, this invention proposes the TADA framework. Randomized prompt enhancement introduces historical categories and random noise terms during the training phase, allowing the model to adapt to complex prompt structures in advance. During testing, distribution-aware fusion dynamically weights LoRA experts based on the Mahalanobis distance between the input image and the category feature distribution, thereby reducing interference from irrelevant experts in the current prediction. Experimental results show that this framework achieves significant performance improvements in task-independent settings and maintains stable performance in standard incremental settings and zero-shot evaluation.

[0087] Example 2: Embodiment 2 of the present invention provides a visual language target detection system oriented towards task-independent incremental methods, comprising: The data acquisition and preprocessing module is configured to acquire continuously arriving target detection task data, including training images, a set of bounding boxes corresponding to the training images, and category labels corresponding to the bounding boxes, and to preprocess the training images. The model and expert database construction module is configured to build a pre-trained open vocabulary object detection model based on an attention mechanism and a category-level low-rank fitting expert database. The model training module is configured to train a pre-trained open vocabulary object detection model using object detection task data, wherein enhanced training prompts containing the current category, historical categories and random noise words are constructed through random prompts. The task-independent reasoning module is configured to perform task-independent reasoning on test images using a pre-trained open-vocabulary object detection model. It dynamically and weightedly fuses category-level experts from a category-level low-rank adaptation expert library based on the Mahalanobis distance between test image features and category distribution to obtain the target bounding box, category name, and confidence score.

[0088] Example 3: Embodiment 3 of the present invention provides a computer-readable storage medium storing a computer program adapted for loading by a processor and executing the steps of the task-independent incremental visual language target detection method as described in Embodiment 1 of the present invention.

[0089] Example 4: Embodiment 4 of the present invention provides a computer device, the device comprising: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the steps of the task-independent incremental visual language target detection method as described in Embodiment 1 of the present invention.

[0090] Example 5: Embodiment 5 of the present invention provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in the task-independent incremental visual language target detection method as described in Embodiment 1 of the present invention.

[0091] The steps and methods involved in Examples 2, 3, 4 and 5 above correspond to those in Example 1. For specific implementation methods, please refer to the relevant description section of Example 1.

[0092] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0093] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data processing device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, an optical medium, or a semiconductor medium, etc.

[0094] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A visual language target detection method oriented towards task-independent incremental approaches, characterized in that, Includes the following steps: Acquire continuously arriving target detection task data, including training images, the set of bounding boxes corresponding to the training images, and the category labels corresponding to the bounding boxes, and preprocess the training images; Construct a pre-trained open-vocabulary target detection model based on attention mechanism and a category-level low-rank fitting expert database; The pre-trained open vocabulary object detection model is trained using object detection task data. In this process, enhanced training prompts containing the current category, historical categories, and random noise words are constructed through random prompts. The pre-trained open vocabulary target detection model is used to perform task-independent inference on the test image. Based on the Mahalanobis distance between the test image features and the category distribution, the category-level experts in the category-level low-rank adaptation expert library are dynamically weighted and fused to obtain the target bounding box, category name and confidence score.

2. The visual language target detection method oriented towards task-independent increments as described in claim 1, characterized in that, Preprocessing operations, including normalization, pixel normalization, and batch processing, are performed on the training images.

3. The visual language target detection method oriented towards task-independent increments as described in claim 1, characterized in that, Pre-trained open-vocabulary object detection models based on attention mechanisms include an image encoder, a text encoder, a cross-modal feature fusion module, a detection decoder, and a detection head.

4. The visual language target detection method oriented towards task-independent increments as described in claim 1, characterized in that, During training, a matching algorithm is used to match the predicted bounding box with the ground truth bounding box. Only the ground truth labeled categories of the training images are used for matching and loss calculation, and the mean of category features and the shared covariance matrix are maintained online.

5. The visual language target detection method oriented towards task-independent incremental methods as described in claim 1, characterized in that, During the task-independent reasoning process of test images using the pre-trained open vocabulary object detection model, the task identity of the test image is not obtained. All learned categories are used to form a unified reasoning prompt. The Mahalanobis distance between the test image features and the mean of each category is calculated. The fusion weight of the category-level low-rank fitting experts is obtained through a normalized exponential function with a temperature coefficient. Then, the expert parameters are dynamically fused and the target bounding box, category name, and confidence score are output.

6. The visual language target detection method oriented towards task-independent increments as described in claim 1, characterized in that, During training, in the parameter update phase, if the training samples contain the true class, the low-rank expert and shared projection matrix corresponding to that true class are updated; the base weights, the main pre-trained weights of the image encoder, and the main pre-trained weights of the text encoder are kept frozen or frozen according to a preset strategy.

7. A visual language target detection system oriented towards task-independent incremental processing, characterized in that, include: The data acquisition and preprocessing module is configured to acquire continuously arriving target detection task data, including training images, a set of bounding boxes corresponding to the training images, and category labels corresponding to the bounding boxes, and to preprocess the training images. The model and expert database construction module is configured to build a pre-trained open vocabulary object detection model based on an attention mechanism and a category-level low-rank fitting expert database. The model training module is configured to train a pre-trained open vocabulary object detection model using object detection task data, wherein enhanced training prompts containing the current category, historical categories and random noise words are constructed through random prompts. The task-independent reasoning module is configured to perform task-independent reasoning on test images using a pre-trained open-vocabulary object detection model. It dynamically and weightedly fuses category-level experts from a category-level low-rank adaptation expert library based on the Mahalanobis distance between test image features and category distribution to obtain the target bounding box, category name, and confidence score.

8. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the task-independent incremental visual language target detection method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed as described in any one of claims 1-6: a visual language target detection method with task-independent incremental approach.

10. A computer device, characterized in that, include: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program that, when executed by the processor, implements the task-independent incremental visual language target detection method as described in any one of claims 1-6.

Images