Target detection method for improving open vocabularies by utilizing vision-text co-occurrence knowledge

Abstract

The invention provides a target detection method for improving open vocabularies by using vision-text co-occurrence knowledge, and relates to the technical field of target detection, comprising the following steps: identifying known category objects in training image data through a to-be-trained target detector; determining a co-occurrence object corresponding to the object of the known category from the training image data, and extracting visual co-occurrence features of the object of the known category and the co-occurrence object; determining co-occurrence category information corresponding to the category label marked by the object of the known category, and extracting text co-occurrence features of the category label and the co-occurrence category information; based on the visual co-occurrence feature and the text co-occurrence feature, determining a pseudo category label corresponding to the co-occurrence object from the co-occurrence category information; and training the target detector by using the known category object and the corresponding category label thereof, and the co-occurrence object and the corresponding pseudo category label thereof. According to the invention, the accuracy of target detection can be obviously improved.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a target detection method for open vocabularies that utilizes visual-text co-occurrence knowledge. Background Technology

[0002] Object detection, a core task in computer vision, aims to locate bounding boxes in images and assign them category labels. In recent years, with the development of deep learning, object detection has made significant progress. However, traditional detectors have a fundamental limitation: they can only recognize a limited number of categories that have appeared in the training data. When faced with a large number of unseen new categories in open-world scenes, traditional detection methods often fail to effectively identify them, making it difficult to meet the needs of practical applications.

[0003] To address the aforementioned issues, the "Open Vocabulary Object Detection (OVOD)" method was proposed. OVOD aims to leverage training data with known categories to enable the detector to identify novel, unseen categories during the inference phase. Its core challenge lies in overcoming the limitations of training labels, allowing the detector to effectively understand and recognize new concepts.

[0004] Existing technologies primarily utilize the image-text alignment capabilities provided by Vision-Language Models (VLMs) to achieve open category detection. These include distillation-based detection methods, feature alignment-based detection methods, prompt-based optimization methods, and detection methods based on Large Language Models (LLMs). While these methods have achieved some success in improving the ability to recognize new categories, they all rely on existing knowledge within the VLM or LLM. This knowledge is often limited to static image-text matching and lacks the ability to model the contextual relationships of real-world scenes. Therefore, the accuracy of category detection results still has room for improvement. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide an improved target detection method for open vocabulary by utilizing visual-text co-occurrence knowledge, which can significantly improve the accuracy of target detection.

[0006] In a first aspect, the present invention provides an improved target detection method for open lexicons using visual-text co-occurrence knowledge, comprising: Acquire training image data; Using the target detector to be trained, identify known category objects in the training image data; The process involves identifying co-occurring objects from training image data that correspond to known category objects, and extracting visual co-occurrence features between known category objects and co-occurring objects; and determining co-occurrence category information corresponding to category labels of known category objects, and extracting textual co-occurrence features between category labels and co-occurrence category information. Co-occurring objects are defined as objects in the training image data that have a co-occurrence relationship with known category objects, and co-occurrence category information is defined as category information that has spatial proximity, functional relevance, or hierarchical relationship with category labels. Based on visual co-occurrence features and text co-occurrence features, the pseudo-category labels corresponding to co-occurring objects are determined from the co-occurrence category information; The target detector is trained using known object categories and their corresponding category labels, as well as co-occurring objects and their corresponding pseudo-category labels. The trained target detector is then used to output the category detection results corresponding to the image data to be processed.

[0007] In one implementation, determining co-occurring objects corresponding to known category objects from training image data, and extracting visual co-occurrence features of known category objects and co-occurring objects, includes: Based on the spatial relationship between known category objects and other objects in the training image data, identify co-occurring objects that have a co-occurrence relationship with known category objects; Merge the bounding boxes of known category objects with the bounding boxes of co-occurring objects to obtain the smallest target bounding box that simultaneously contains both known category objects and co-occurring objects. Visual co-occurrence features of known category objects and co-occurring objects are extracted based on the target bounding box using a visual encoder.

[0008] In one implementation, based on the spatial relationship between known category objects and other objects in the training image data, co-occurring objects that have a co-occurrence relationship with known category objects are determined, including: For objects other than known category objects in the training image data, determine the intersection-union ratio and center distance between the object and known category objects, and perform weighted fusion on the intersection-union ratio and center distance to obtain the co-occurrence relationship score between the object and known category objects. The object corresponding to the highest co-occurrence relationship score is identified as a co-occurring object that has a co-occurrence relationship with objects of the known category.

[0009] In one implementation, the large language model embeds spatial proximity query commands, functional relevance query commands, and hierarchical relationship query commands; it determines the co-occurrence category information corresponding to the category labels of known category objects, and extracts the text co-occurrence features of the category labels and co-occurrence category information, including: Using the spatial proximity query command embedded in the large language model, co-occurrence category information that has spatial proximity with the category labels of known category objects is determined; and using the functional relevance query command embedded in the large language model, co-occurrence category information that has functional relevance with the category labels of known category objects is determined; and using the hierarchical relationship query command embedded in the large language model, co-occurrence category information that has hierarchical relationship with the category labels of known category objects is determined. For any co-occurrence category information, a text co-occurrence description is constructed based on the co-occurrence category information and the category label, and the text co-occurrence description is encoded into a text co-occurrence feature by a text encoder.

[0010] In one implementation, based on visual co-occurrence features and text co-occurrence features, pseudo-category labels corresponding to co-occurring objects are determined from co-occurrence category information, including: For any text co-occurrence feature, determine the cosine similarity between the text co-occurrence feature and the visual co-occurrence feature; The cosine similarity is converted into a probability value. When the maximum probability value is greater than a preset threshold, the co-occurrence category information of the text co-occurrence feature corresponding to the maximum probability value is determined as the pseudo-category label of the co-occurring object.

[0011] In one implementation, the target detector is trained using known categories of objects and their corresponding category labels, as well as co-occurring objects and their corresponding pseudo-category labels, including: By using a pre-built loss function, the training loss value is determined based on the category detection results output by the object detector for objects in the training image data, and their corresponding category labels or pseudo-category labels, in order to adjust the parameters of the object detector; The loss function includes at least a pseudo-label classification loss function.

[0012] In one implementation, the expression for the pseudo-label classification loss function is as follows: ; ; This is the loss value; This represents the number of pseudo-category labels; For category detection results; For co-occurring objects Pseudo-category tags; For co-occurring objects Features; As an indicator function, in the category detection results, the first... Categories and pseudo-categorization labels In the case of a match, its value is 1, and it is the [number]th [rank] in the category detection results. Categories and pseudo-categorization labels In the case of a mismatch, its value is 0; For co-occurring objects Belongs to the The probability of each category; For the first Text embeddings for each category; For the category detection results, the first Text embeddings for each category; Temperature hyperparameters are used to adjust the sharpness of the probability distribution.

[0013] Secondly, the present invention also provides a target detection device for open vocabulary that utilizes visual-text co-occurrence knowledge, comprising: The data acquisition module is used to acquire training image data; The object recognition module is used to identify known category objects in the training image data using the target detector to be trained; The co-occurrence feature extraction module is used to determine co-occurring objects corresponding to known category objects from training image data, and extract the visual co-occurrence features of known category objects and co-occurring objects; and to determine the co-occurrence category information corresponding to the category labels of known category objects, and extract the textual co-occurrence features of category labels and co-occurrence category information; wherein, co-occurring objects are: objects in training image data that have a co-occurrence relationship with known category objects, and co-occurrence category information is: category information that has spatial proximity, functional relevance or hierarchical relationship with category labels; The pseudo-category label determination module is used to determine the pseudo-category label corresponding to the co-occurring objects from the co-occurrence category information based on visual co-occurrence features and text co-occurrence features; The training module is used to train the target detector using known object categories and their corresponding category labels, co-occurring objects and their corresponding pseudo-category labels. The trained target detector is used to output the category detection results corresponding to the image data to be processed.

[0014] Thirdly, the present invention also provides an electronic device including a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement any of the methods provided in the first aspect.

[0015] Fourthly, the present invention also provides a computer-readable storage medium storing computer-executable instructions, which, when invoked and executed by a processor, cause the processor to implement any of the methods provided in the first aspect.

[0016] This invention provides an improved target detection method for open vocabularies using visual-text co-occurrence knowledge. First, training image data is acquired, and a target detector to be trained identifies known category objects in the training image data. Then, co-occurring objects corresponding to the known category objects are determined from the training image data, and visual co-occurrence features of the known category objects and co-occurring objects are extracted. Next, co-occurrence category information corresponding to the category labels of the known category objects is determined, and textual co-occurrence features of the category labels and co-occurrence category information are extracted. Co-occurring objects are those objects in the training image data that have a co-occurrence relationship with known category objects, and co-occurrence category information is category information that has spatial proximity, functional relevance, or hierarchical relationship with the category labels. Then, based on the visual and textual co-occurrence features, pseudo-category labels corresponding to the co-occurring objects are determined from the co-occurrence category information. Finally, the target detector is trained using the known category objects and their corresponding category labels, and the co-occurring objects and their corresponding pseudo-category labels. The trained target detector outputs the category detection results corresponding to the image data to be processed. The above method identifies co-occurring objects and extracts their corresponding visual co-occurrence features for known categories of objects in training image data, determines the co-occurrence category information corresponding to their labeled category tags and extracts the corresponding textual co-occurrence features, and then determines the pseudo-category labels corresponding to the co-occurring objects for training the object detector. This invention is not limited to the knowledge inside the model, but actively extracts and utilizes external, multimodal contextual information from the real-world scene as pseudo-category labels to make up for the shortcomings of existing technologies in contextual reasoning, thereby effectively identifying and locating new categories of objects in image data and significantly improving the performance of object detection.

[0017] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0018] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0019] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1A technical flowchart of a conventional VLM-based OVOD method provided in an embodiment of the present invention; Figure 2 This is a flowchart illustrating an improved target detection method for open vocabularies using visual-text co-occurrence knowledge, provided as an embodiment of the present invention. Figure 3 This is a technical framework diagram of an open vocabulary target detection method provided by an embodiment of the present invention, which utilizes visual-text co-occurrence knowledge. Figure 4 This is a schematic diagram of the structure of a target detection device for open vocabulary that utilizes visual-text co-occurrence knowledge, provided in an embodiment of the present invention. Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Currently, existing technologies primarily achieve openness detection through the image-text alignment capabilities provided by the Vision-Language Model (VLM). (1) Detection methods based on distillation: such as the ViLD (Vision-Language Model with Open-Vocabulary Detection) method, which transfers the image-text alignment capability in VLM to the detector through knowledge distillation; HierKD (Hierarchical Knowledge Distillation), RKD (Relational Knowledge Distillation) and other methods further adopt hierarchical distillation or regional distillation to improve the alignment effect.

[0023] (2) Feature alignment-based methods: such as VLDet (Vision-Language Detection) and Detic (DEtecting objects with Transformer and Image Captions), which align the detector output features with the text embedding of VLM to support open vocabulary detection.

[0024] (3) Prompt-based optimization methods: such as DetPro (Detection with Prototypical Learning) and PromptDet (Prompt-based Detection), which fine-tune the text embedding of VLM to make it more suitable for the regional features of the detector.

[0025] (4) LLM has also been introduced into OVOD as a static semantic knowledge base. For example, CupL uses LLM to generate rich category descriptions; CaFo uses GPT-3 to build semantic text to improve the alignment capability of CLIP (Contrastive Language–ImagePre-training); SHiNe uses LLM to build hierarchical category relationships; MM-OVOD (Multi-Modal Open-Vocabulary Object Detection) and DVDet further use LLM to generate fine-grained or diverse category descriptions.

[0026] See Figure 1 The technical flowchart shown is a technical flowchart of an existing VLM-based OVOD method, which mainly includes the following: (1) Class relationship modeling: Given a set of classes : Target categories labeled in the training set; New category set Categories that are not labeled in the training set but may appear during the inference phase; open category set. This includes known categories, new categories, and arbitrary natural language descriptions. and They are mutually exclusive, and together they form part of an open set.

[0027] (2) Typical technical architecture of OVOD based on VLM: Visual encoder Extracting features from candidate regions in an image .

[0028] Text encoder Transform category labels or text descriptions into semantic vectors. .

[0029] Visual-linguistic feature alignment: By sharing semantic space constraints, regional features... Text features Global image features Alignment.

[0030] Detection and Classification: Used to predict bounding boxes , After projecting the region features onto the semantic space, classification is performed, and the category is output. .

[0031] (3) Working method: During the training phase, the detector is supervised by known category labels and open text descriptions to learn the correspondence between visual features and linguistic features. During the inference phase, for the input image, the detector outputs bounding boxes and corresponding categories, which can not only identify known categories, but also identify new categories not seen in the training set.

[0032] Analysis reveals that while existing open-vocabulary object detection technologies based on vision-language models can overcome the limitations of closed category sets and identify new, unseen categories to some extent, they still have the following objective drawbacks: (1) Lack of scene context modeling: Existing methods mainly rely on the image-text alignment capability of VLM to complete the detection, failing to effectively utilize the co-occurrence relationships of targets that are widely present in real scenes (such as "fork" often appearing at the same time as "knife" and "plate"). This results in the detector lacking contextual constraints when identifying new categories, and the detection accuracy is limited. This invention can make up for the shortcomings of existing detectors in utilizing contextual information by introducing a co-occurrence relationship modeling mechanism.

[0033] (2) Lack of effective verification of semantic knowledge: Some methods use large language models (LLM) to generate category descriptions or hierarchical semantic information, but such texts often contain ambiguous or redundant content, or even semantic noise, which can easily affect the accuracy of detection results. This invention uses a dual mechanism of "visual co-occurrence extraction - LLM verification" to ensure the accuracy and reliability of semantic relationships, thereby reducing noise interference.

[0034] (3) Insufficient stability of new category detection: Due to the lack of direct sample supervision of new categories during the training phase, existing detectors often have low prediction confidence for new categories during inference, which easily leads to false detections and false negatives. This invention introduces pseudo-category labels based on co-occurrence relationships, enabling the detector to obtain indirect supervision of new categories during the training phase, thereby improving the stability of new category recognition.

[0035] (4) Limited versatility of the method: Some existing methods require modification of the detector structure or deep coupling, making them difficult to apply directly to different detection frameworks and limiting practical application. This invention adopts a plug-and-play technique, which can enhance any existing VLM-based detector without changing the detector architecture, and has stronger compatibility and scalability.

[0036] Based on this, the present invention provides an improved target detection method for open vocabularies by utilizing visual-text co-occurrence knowledge, which can solve the problems of lack of scene context modeling, lack of effective verification of semantic knowledge, insufficient stability of new category detection and limited method universality in the existing technology, thereby significantly improving the accuracy of target detection.

[0037] To facilitate understanding of this embodiment, a detailed description of an improved open-vocabulary target detection method utilizing visual-text co-occurrence knowledge, as disclosed in this embodiment of the invention, will be provided first. (See [link to relevant documentation]). Figure 2 The diagram shows a flowchart of an open-vocabulary target detection method that utilizes visual-text co-occurrence knowledge. The method mainly includes the following steps S202 to S210: Step S202: Obtain training image data.

[0038] Step S204: Identify known category objects in the training image data using the target detector to be trained. The target detector can be a neural network-based target detection model, such as ViLD, and the known category objects are those already labeled with category tags. In one implementation, the target detector's workflow is as follows: extract features from the training image data to identify the objects contained within it and generate corresponding bounding boxes; further, detect the probability that the objects within the bounding boxes belong to each category, using this probability as the category detection result corresponding to the training image data.

[0039] Step S206: Determine the co-occurring objects corresponding to known category objects from the training image data, and extract the visual co-occurrence features of known category objects and co-occurring objects; and determine the co-occurrence category information corresponding to the category labels of known category objects, and extract the textual co-occurrence features of category labels and co-occurrence category information.

[0040] Co-occurring objects are defined as objects in the training image data that co-occur with objects of known categories. Whether a known category object co-occurs with other objects can be determined through spatial relationship evaluation. Visual co-occurrence features can be understood as features corresponding to the bounding boxes of both known category objects and co-occurring objects. Co-occurrence category information refers to category information that exhibits spatial proximity, functional relevance, or hierarchical relationship with the category label. Spatial proximity can be understood as objects frequently appearing simultaneously in space (e.g., keyboard and mouse); functional relevance can be understood as objects with complementary or identical functions often appearing simultaneously (e.g., motorcycle and bicycle); hierarchical relationship can be understood as objects within the same broader category exhibiting co-occurrence, such as forks and bowls under the tableware category. Text co-occurrence features can be understood as features corresponding to text that simultaneously contains both category labels and co-occurrence category information.

[0041] Step S208: Based on visual co-occurrence features and text co-occurrence features, determine the pseudo-category label corresponding to the co-occurring object from the co-occurrence category information.

[0042] In one example, visual co-occurrence features and text co-occurrence features can be aligned, and the visual and text co-occurrence relationships can be matched by maximizing the cosine similarity between the visual and text co-occurrence features. That is, the pseudo-category label corresponding to the co-occurring object can be determined from the co-occurrence category information using the cosine similarity between the visual and text co-occurrence features.

[0043] Step S210: The target detector is trained using known category objects and their corresponding category labels, co-occurring objects and their corresponding pseudo category labels. The trained target detector is used to output the category detection results corresponding to the image data to be processed. The category detection results include the probability that the objects displayed in the image data to be processed belong to each category.

[0044] In one implementation, embodiments of the present invention incorporate high-confidence pseudo-class labels into the training process of the OVOD framework. Specifically, an additional pseudo-class label classification loss is introduced to encourage the visual embeddings of co-occurring objects to align with the textual embeddings of their pseudo-class labels, thereby associating ambiguous visual regions with semantically relevant co-occurrence categories. This effectively bridges the gap between known and new concepts, significantly improving the detection performance of the object detector.

[0045] The present invention provides an improved object detection method for open vocabularies using visual-text co-occurrence knowledge. For known categories of objects in training image data, it identifies co-occurring objects and extracts corresponding visual co-occurrence features, determines the co-occurrence category information corresponding to the labeled category and extracts corresponding text co-occurrence features, and then determines pseudo-category labels corresponding to the co-occurring objects. These pseudo-category labels are then used to train the object detector. The present invention is not limited to internal model knowledge but actively extracts and utilizes external, multimodal contextual information from real-world scenes as pseudo-category labels to compensate for the shortcomings of existing technologies in contextual reasoning. This enables effective identification and localization of new categories of objects in image data, significantly improving the performance of object detection.

[0046] Existing methods, such as Open Vocabulary Object Detection (OVOD) based on Visual-Language Models (VLMs), primarily align visual and text embeddings by distilling knowledge from pre-trained VLMs into the object detector, thereby achieving generalization to unknown categories. However, these methods heavily rely on the internal knowledge of the VLM, which has limited understanding of context, resulting in insufficient generalization ability when faced with new categories.

[0047] This invention aims to address this technical challenge by proposing a plug-and-play framework (CODet) that enhances OVOD by integrating object co-occurrence knowledge—an external contextual knowledge. The fundamental difference between this invention and existing technologies lies in its proactive extraction and utilization of external, multimodal contextual information from real-world scenes, rather than being limited to internal model knowledge, to compensate for the shortcomings of existing VLMs in contextual reasoning. In this way, this invention can more effectively identify and locate new object categories, achieving a significant performance improvement.

[0048] For ease of understanding, this invention provides a specific implementation of an open-vocabulary target detection method that utilizes visual-text co-occurrence knowledge. Taking the Backbone+RPN model as an example, see [link to relevant documentation]. Figure 3 The diagram shown illustrates a technical framework for improving open-vocabulary object detection methods using visual-text co-occurrence knowledge, comprising: Step 1: Obtain training image data.

[0049] Step 2: Using the Backbone+RPN model, identify objects of known categories in the training image data. and its bounding box, the annotation of the known category object. Category tags .

[0050] Step 3: Identify objects of known categories from the training image data. Corresponding co-occurring objects Extract objects of known categories. and co-occurring objects Visual co-occurrence features.

[0051] In one implementation, this can be achieved by analyzing objects of known categories. Compared with other objects in the training image data Spatial relationships between objects, finding those of known categories. Co-occurring objects with co-occurrence relationship Extract objects of known categories. and co-occurring objects Visual co-occurrence features Specifically, this includes: Step 3.1, based on known object categories Compared with other objects in the training image data The spatial relationship between them determines the relationship with known categories of objects. Co-occurring objects with co-occurrence relationship Specifically, this includes: (1) For objects in the training image data other than those of known categories Other objects Determine the object Objects of known categories The intersection-over-union ratio (CUIR) and center-point distance between objects are weighted and fused to obtain the object's data. Objects of known categories The co-occurrence relationship score between them.

[0052] In practical implementation, a scoring function is designed. This is used to quantify the co-occurrence relationship between two objects. The scoring function considers both their intersection-over-union (IoU) and centroid distance, i.e.: ; ; ; in, and These are hyperparameters used to balance IoU and the distance to the center point. Objects of known categories and objects The distance between the center points It is the diagonal length of the training image data. , To train the width and length of the image data, , For objects of known categories The center pixel coordinates in the training image data , For objects The pixel coordinates of the center point in the training image data.

[0053] (2) The object corresponding to the highest co-occurrence score It was determined to be an object of a known category. Co-occurring objects with co-occurrence relationship ,Right now: .

[0054] In another implementation, co-occurrence quantification: In addition to using a weighted combination of IoU and center point distance to quantify co-occurrence relationships, other metrics can be used, such as: (1) Relative size ratio: measuring the ratio of the bounding box areas of two objects. (2) Relative position encoding: encoding the relative position of co-occurring objects with respect to known objects (e.g., "left side", "above") as features. (3) Attention mechanism: using an attention mechanism to dynamically learn the importance of different co-occurring objects.

[0055] Step 3.2: Merge the bounding boxes of known category objects and co-occurring objects to obtain the smallest target bounding box that simultaneously contains both known category objects and co-occurring objects. In specific implementations, the known category objects... and its co-occurring objects Merge into a minimum target bounding box that contains both. .

[0056] Step 3.3 involves extracting visual co-occurrence features of known category objects and co-occurring objects based on the target bounding box using a visual encoder. In practice, this is achieved using the visual encoder of a VLM (Virtual Machine Library). Extract its visual co-occurrence features .

[0057] Step 4: Determine the co-occurrence category information corresponding to the category labels of known category objects, and extract the text co-occurrence features of the category labels and co-occurrence category information.

[0058] To ensure that the visually extracted co-occurrence relationships have semantic meaning, this embodiment of the invention introduces a large language model (LLM) as an external knowledge base for verification. For each category label... Three complementary query templates were designed to ask questions of the LLM to obtain co-occurrence category information with spatial proximity, functional relevance, or hierarchical relationship. Extract category labels Co-occurrence category information Text co-occurrence features The query templates include spatial proximity query commands, functional relevance query commands, and hierarchical relationship query commands.

[0059] The specific implementation includes the following steps: Step 4.1: Using the spatial proximity query command embedded in the large language model, determine the co-occurrence category information that has spatial proximity with the category labels of known category objects.

[0060] Among them, spatial proximity query: objects often appear simultaneously in space (e.g., keyboard and mouse). To capture this, embodiments of the present invention use the following command (i.e., spatial proximity query command) to query the LLM: "Co-occurrence category information is directly generated based on spatial adjacency observed in real-world scenarios." .

[0061] Step 4.2: Using the functional relevance query command embedded in the large language model, determine the co-occurrence category information that has functional relevance to the category labels of known category objects.

[0062] Among them, functional relevance query: Objects with complementary or identical functions often appear together (e.g., "motorcycle" and "bicycle"). This embodiment of the invention uses the following command (i.e., the functional relevance query command) to query the LLM: This utilizes the functional dependencies of objects to infer coexistence relationships beyond spatial proximity, thereby generating co-occurrence category information. .

[0063] Step 4.3: Using the hierarchical relationship query command embedded in the large language model, determine the co-occurrence category information that has a hierarchical relationship with the category labels of known category objects.

[0064] In this context, hierarchical relationship queries involve objects within the same broader category (e.g., forks and bowls under cutlery) exhibiting co-occurrence. Embodiments of this invention use the following command (i.e., a hierarchical relationship query command) to query the LLM: This query utilizes hierarchical semantics to identify sibling categories within a classification system, thereby generating co-occurrence category information. .

[0065] In another implementation, text co-occurrence verification: In addition to using LLM for verification, other external knowledge bases can be utilized, such as: (1) Knowledge graph: directly querying the co-occurrence or hierarchical relationships between objects from existing knowledge graphs. (2) Text corpus: inferring semantic relationships between objects by analyzing word frequencies and co-occurrence patterns in a large-scale text corpus.

[0066] Step 4.4: For any co-occurrence category information, construct a text co-occurrence description based on the co-occurrence category information and the category label, and encode the text co-occurrence description into a text co-occurrence feature using a text encoder.

[0067] In practical implementation, through these queries, embodiments of the present invention obtain a set of potential co-occurrence categories. Then, for each co-occurrence category information... and category labels Construct a text co-occurrence description, for example: A photo of and Finally, the VLM text encoder is used. Encode these descriptions into text co-occurrence features .

[0068] .

[0069] Step 5: Based on visual co-occurrence features and text co-occurrence features, determine the pseudo-category labels corresponding to the co-occurring objects from the co-occurrence category information, specifically including: Step 5.1, for any text co-occurrence feature Determine the co-occurrence features of the text. Co-occurrence features with vision Cosine similarity between .

[0070] In practical implementation, when obtaining visual co-occurrence features Co-occurrence features with text Subsequently, embodiments of the present invention utilize the alignment capabilities of VLM pre-training to match visual and textual co-occurrence relationships by maximizing their cosine similarity: .

[0071] Step 5.2: Convert cosine similarity into probability values. If the maximum probability value is greater than a preset threshold, determine the co-occurrence category information of the text co-occurrence feature corresponding to the maximum probability value as the pseudo-category label of the co-occurring object.

[0072] In the actual implementation, the cosine similarity is converted into a probability value through a sigmoid function. Used to determine co-occurrence category information Is it a co-occurring object? Pseudo-category labels. Only when the probability value is higher than a preset threshold... Only then will the co-occurrence category information with the highest probability value be included. As co-occurring objects The pseudo-category label. The expression for the Sigmoid function is as follows: .

[0073] Step 6: Train the target detector using known object categories and their corresponding category labels, co-occurring objects and their corresponding pseudo-category labels. Specifically, this includes: determining the training loss value based on the category detection results output by the target detector for objects in the training image data and their corresponding category labels or pseudo-category labels using a pre-constructed loss function, in order to adjust the parameters of the target detector; wherein the loss function includes at least a pseudo-label classification loss function.

[0074] In practical implementation, by extracting or aligning co-occurrence relationships, this embodiment of the invention obtains a set of high-confidence co-occurring object bounding boxes. and its associated pseudo-category labels , For co-occurring objects The number of pseudo-category labels (also known as the number of pseudo-category labels), these high-confidence pseudo-category labels { This is injected into the training process of the OVOD framework. Specifically, this embodiment of the invention introduces an additional pseudo-label classification loss. Its purpose is to encourage co-occurring objects Visual embeddings and their pseudo-category labels Text embedding alignment effectively bridges the gap between known and new concepts by associating ambiguous visual regions with semantically relevant co-occurrence categories.

[0075] The expression for the pseudo-label classification loss function is as follows: ; ; This is the loss value; This represents the number of pseudo-category labels; For category detection results; For co-occurring objects Pseudo-category tags; For co-occurring objects Features; As an indicator function, in the category detection results, the first... Categories and pseudo-categorization labels In the case of a match, its value is 1, and it is the [number]th [rank] in the category detection results. Categories and pseudo-categorization labels In the case of a mismatch, its value is 0; For co-occurring objects Belongs to the The probability of each category; For the first Text embeddings for each category; For the category detection results, the first Text embeddings for each category; Temperature hyperparameters are used to adjust the sharpness of the probability distribution.

[0076] In another implementation, in addition to using co-occurrence relationships as pseudo-labels for training, knowledge can be injected into the model in other ways: (1) Feature fusion: Visual co-occurrence features and text co-occurrence features are directly fused at different levels of the model. (2) Constraint loss: A new loss function is designed to directly constrain the model so that co-occurring objects are close to each other in the feature space.

[0077] In summary, the core idea of ​​the present invention is to extract visual co-occurrence patterns from images and verify them using the semantic knowledge of large language models (LLMs). These external, multimodal aligned co-occurrence knowledge are then injected as pseudo-labels into the training of the Open Vocabulary Object Detection (OVOD) model, thereby compensating for the shortcomings of existing models in contextual understanding and improving their generalization ability to new categories. The core technical points of the present invention are as follows: (1) A cross-modal alignment method combining visual co-occurrence patterns with textual dependencies verified by LLMs is proposed to generate high-quality pseudo-labels, thereby injecting external contextual knowledge into OVOD. (2) A plug-and-play framework is designed to seamlessly enhance existing VLM-based object detectors without any architectural modifications. (3) A scoring function for quantifying object co-occurrence relationships is proposed, which comprehensively considers spatial proximity and overlap to effectively identify co-occurring objects. (4) A set of complementary LLM query templates (including spatial, functional, and hierarchical relationships) is designed to systematically obtain semantically meaningful co-occurrence categories.

[0078] Based on this, the embodiments of the present invention have at least the following characteristics: (i) Improving generalization ability for new categories: Existing methods rely heavily on the internal knowledge of VLM, resulting in limited generalization ability when facing new categories. This invention introduces external co-occurrence knowledge to provide the model with richer contextual information, enabling the model to infer the category of unknown objects based on the context of known objects, thereby significantly improving the detection performance for new categories, with an AP value improvement of up to 2.3%.

[0079] (ii) Enhanced cross-modal alignment: By aligning visually extracted co-occurrence patterns with text dependencies verified by LLM, this embodiment of the invention effectively solves the visual-text context misalignment problem existing in existing methods. This cross-modal fusion mechanism enables the model to better understand complex scenes and establish more accurate visual-text associations, thereby achieving more robust detection.

[0080] (III) Plug and play, strong compatibility: The embodiments of the present invention are plug and play frameworks that do not require any modification to the architecture of existing VLM-based target detectors. It can be seamlessly integrated into a variety of mainstream detectors (such as ViLD and DetPro) and continuously bring performance improvements, which greatly reduces the difficulty and cost of technical implementation.

[0081] (iv) Avoiding false associations: By using the semantic knowledge of LLM for verification, the embodiments of the present invention can effectively filter out visually accidental but semantically irrelevant co-occurrence patterns, ensuring that the co-occurrence knowledge learned by the model is real and meaningful, avoiding the learning of false associations, thereby improving the robustness of the model.

[0082] Based on the foregoing embodiments, this invention provides an improved target detection device for open vocabularies using visual-text co-occurrence knowledge. (See also...) Figure 4 The diagram shows a structural schematic of a target detection device that utilizes visual-text co-occurrence knowledge to improve open vocabulary. The device mainly includes the following components: Data acquisition module 402 is used to acquire training image data; The object recognition module 404 is used to recognize known category objects in the training image data through the target detector to be trained; The co-occurrence feature extraction module 406 is used to determine co-occurring objects corresponding to known category objects from training image data, and extract the visual co-occurrence features of known category objects and co-occurring objects; and to determine the co-occurrence category information corresponding to the category labels of known category objects, and extract the textual co-occurrence features of category labels and co-occurrence category information; wherein, co-occurring objects are: objects in training image data that have a co-occurrence relationship with known category objects, and co-occurrence category information is: category information that has spatial proximity, functional relevance or hierarchical relationship with category labels; The pseudo-category label determination module 408 is used to determine the pseudo-category label corresponding to the co-occurring object from the co-occurrence category information based on visual co-occurrence features and text co-occurrence features; The training module 410 is used to train the target detector using known categories of objects and their corresponding category labels, co-occurring objects and their corresponding pseudo-category labels. The trained target detector is used to output the category detection results corresponding to the image data to be processed.

[0083] The object detection device for open vocabulary improved by utilizing visual-text co-occurrence knowledge provided in this invention identifies co-occurring objects and extracts corresponding visual co-occurrence features for known categories of objects in training image data, determines the co-occurrence category information corresponding to the labeled category and extracts corresponding text co-occurrence features, and then determines the pseudo-category labels corresponding to the co-occurring objects for training the object detector. This invention is not limited to internal model knowledge, but actively extracts and utilizes external, multimodal contextual information from real-world scenes as pseudo-category labels to compensate for the shortcomings of existing technologies in contextual reasoning, thereby effectively identifying and locating new categories of objects in image data and significantly improving the performance of object detection.

[0084] In one implementation, the co-occurrence feature extraction module 406 is specifically used for: Based on the spatial relationship between known category objects and other objects in the training image data, identify co-occurring objects that have a co-occurrence relationship with known category objects; Merge the bounding boxes of known category objects with the bounding boxes of co-occurring objects to obtain the smallest target bounding box that simultaneously contains both known category objects and co-occurring objects. Visual co-occurrence features of known category objects and co-occurring objects are extracted based on the target bounding box using a visual encoder.

[0085] In one implementation, the co-occurrence feature extraction module 406 is specifically used for: For objects other than known category objects in the training image data, determine the intersection-union ratio and center distance between the object and known category objects, and perform weighted fusion on the intersection-union ratio and center distance to obtain the co-occurrence relationship score between the object and known category objects. The object corresponding to the highest co-occurrence relationship score is identified as a co-occurring object that has a co-occurrence relationship with objects of the known category.

[0086] In one implementation, the large language model embeds spatial proximity query commands, functional relevance query commands, and hierarchical relationship query commands; the co-occurrence feature extraction module 406 is specifically used for: Using the spatial proximity query command embedded in the large language model, co-occurrence category information that has spatial proximity with the category labels of known category objects is determined; and using the functional relevance query command embedded in the large language model, co-occurrence category information that has functional relevance with the category labels of known category objects is determined; and using the hierarchical relationship query command embedded in the large language model, co-occurrence category information that has hierarchical relationship with the category labels of known category objects is determined. For any co-occurrence category information, a text co-occurrence description is constructed based on the co-occurrence category information and the category label, and the text co-occurrence description is encoded into a text co-occurrence feature by a text encoder.

[0087] In one implementation, the pseudo-category label determination module 408 is specifically used for: For any text co-occurrence feature, determine the cosine similarity between the text co-occurrence feature and the visual co-occurrence feature; The cosine similarity is converted into a probability value. When the maximum probability value is greater than a preset threshold, the co-occurrence category information of the text co-occurrence feature corresponding to the maximum probability value is determined as the pseudo-category label of the co-occurring object.

[0088] In one implementation, the training module 410 is specifically used for: By using a pre-built loss function, the training loss value is determined based on the category detection results output by the object detector for objects in the training image data, and their corresponding category labels or pseudo-category labels, in order to adjust the parameters of the object detector; The loss function includes at least a pseudo-label classification loss function.

[0089] In one implementation, the expression for the pseudo-label classification loss function is as follows: ; ; This is the loss value; This represents the number of pseudo-category labels; For category detection results; For co-occurring objects Pseudo-category tags; For co-occurring objects Features; As an indicator function, in the category detection results, the first... Categories and pseudo-categorization labels In the case of a match, its value is 1, and it is the [number]th [rank] in the category detection results. Categories and pseudo-categorization labels In the case of a mismatch, its value is 0; For co-occurring objects Belongs to the The probability of each category; For the first Text embeddings for each category; For the category detection results, the first Text embeddings for each category; Temperature hyperparameters are used to adjust the sharpness of the probability distribution.

[0090] The device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0091] This invention provides an electronic device, specifically, the electronic device includes a processor and a memory; the memory stores a computer program, which, when run by the processor, executes the method described in any of the above embodiments.

[0092] Figure 5The present invention provides a schematic diagram of the structure of an electronic device 100, which includes a processor 50, a memory 51, a bus 52 and a communication interface 53. The processor 50, the communication interface 53 and the memory 51 are connected through the bus 52. The processor 50 is used to execute executable modules, such as computer programs, stored in the memory 51.

[0093] The memory 51 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 53 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0094] Bus 52 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0095] The memory 51 is used to store programs. After receiving an execution instruction, the processor 50 executes the programs. The method executed by the device for defining the flow process disclosed in any of the foregoing embodiments of the present invention can be applied to the processor 50 or implemented by the processor 50.

[0096] Processor 50 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 50 or by instructions in software form. Processor 50 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 51. The processor 50 reads the information in memory 51 and, in conjunction with its hardware, completes the steps of the above method.

[0097] The computer program product of the readable storage medium provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the foregoing method embodiments. For specific implementation, please refer to the foregoing method embodiments, which will not be repeated here.

[0098] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that co-occurs with the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0099] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for target detection using visual-text co-occurrence knowledge to improve open vocabulary, characterized in that, The method comprises: acquiring training image data; identifying a known category object in the training image data by a target detector to be trained; determining a co-occurring object corresponding to the known category object from the training image data, and extracting visual co-occurrence features of the known category object and the co-occurring object; determining co-occurrence category information corresponding to a category label of the known category object, and extracting text co-occurrence features of the category label and the co-occurrence category information; wherein the co-occurring object is an object that has a co-occurrence relationship with the known category object in the training image data, and the co-occurrence category information is category information that has spatial proximity, functional correlation or hierarchical relationship with the category label; determining a pseudo category label corresponding to the co-occurring object from the co-occurrence category information based on the visual co-occurrence features and the text co-occurrence features; training the target detector using the known category object and the corresponding category label, the co-occurring object and the corresponding pseudo category label, so that the trained target detector is used to output a category detection result corresponding to to-be-processed image data. 2.The target detection method of improving open vocabulary by visual-text co-occurrence knowledge according to claim 1, wherein, The method for determining a co-occurring object corresponding to the known category object from the training image data and extracting visual co-occurrence features of the known category object and the co-occurring object comprises: determining a co-occurring object having a co-occurrence relationship with the known category object based on a spatial relationship between the known category object and other objects in the training image data; merging a bounding box of the known category object and a bounding box of the co-occurring object to obtain a target bounding box that contains the known category object and the co-occurring object at the same time and is the smallest; extracting visual co-occurrence features of the known category object and the co-occurring object based on the target bounding box by a visual encoder. 3.The target detection method of improving open vocabulary by visual-text co-occurrence knowledge according to claim 2, characterized in that, The method for determining a co-occurring object having a co-occurrence relationship with the known category object based on a spatial relationship between the known category object and other objects in the training image data comprises: for an object other than the known category object in the training image data, determining an intersection-over-union and a center point distance between the object and the known category object, and performing weighted fusion on the intersection-over-union and the center point distance to obtain a co-occurrence relationship score value between the object and the known category object; determining an object corresponding to the highest co-occurrence relationship score value as the co-occurring object having a co-occurrence relationship with the known category object. 4.The target detection method of improving open vocabulary by visual-text co-occurrence knowledge according to claim 1, wherein, The large language model is embedded with a spatial proximity query command, a functional correlation query command and a hierarchical relationship query command. The method for determining co-occurrence category information corresponding to a category label of the known category object and extracting text co-occurrence features of the category label and the co-occurrence category information comprises: determine, through the spatial proximity query command embedded in the large language model, co-occurrence category information that has spatial proximity with the category label of the known category object, and determine, through the functional correlation query command embedded in the large language model, co-occurrence category information that has functional correlation with the category label of the known category object, and determine, through the hierarchical relationship query command embedded in the large language model, co-occurrence category information that has hierarchical relationship with the category label of the known category object; for any co-occurrence category information, construct a text co-occurrence description for the co-occurrence category information and the category label, and encode the text co-occurrence description into text co-occurrence features through a text encoder.

5. The target detection method for improving open vocabulary using visual-text co-occurrence knowledge according to claim 1, wherein, determine, based on the visual co-occurrence features and the text co-occurrence features, a pseudo category label corresponding to the co-occurrence object from the co-occurrence category information, including: for any text co-occurrence feature, determine the cosine similarity between the text co-occurrence feature and the visual co-occurrence feature; convert the cosine similarity into a probability value, and in a case where the maximum probability value is greater than a preset threshold, determine the co-occurrence category information to which the text co-occurrence feature corresponding to the maximum probability value belongs as the pseudo category label corresponding to the co-occurrence object. 6.The method of claim 1, wherein, train the target detector using the known category object and the corresponding category label, the co-occurrence object and the corresponding pseudo category label, including: determine, through a pre-constructed loss function, a training loss value based on the category detection result output by the target detector for the object in the training image data and the corresponding category label or pseudo category label, to adjust the parameters of the target detector; wherein the loss function at least includes a pseudo label classification loss function.

7. The target detection method for improving open vocabulary using visual-text co-occurrence knowledge according to claim 6, wherein, The expression of the pseudo label classification loss function is as follows: ； ； This is the loss value; This represents the number of pseudo-category labels; For category detection results; For co-occurring objects Pseudo-category tags; For co-occurring objects Features; As an indicator function, in the category detection results, the first... Categories and pseudo-categorization labels In the case of a match, its value is 1, and it is the [number]th [rank] in the category detection results. Categories and pseudo-categorization labels In the case of a mismatch, its value is 0; For co-occurring objects Belongs to the The probability of each category; For the first Text embeddings for each category; For the first category detection result Text embeddings for each category; Temperature hyperparameters used to adjust the sharpness of the probability distribution.

8. A target detection device for open vocabulary that utilizes visual-text co-occurrence knowledge, characterized in that, including: a data acquisition module configured to acquire training image data; an object recognition module configured to recognize a known category object in the training image data through a target detector to be trained; a co-occurrence feature extraction module configured to determine a co-occurrence object corresponding to the known category object from the training image data, and extract visual co-occurrence features of the known category object and the co-occurrence object; and determine co-occurrence category information corresponding to the category label of the known category object, and extract text co-occurrence features of the category label and the co-occurrence category information; wherein the co-occurrence object is an object in the training image data that has a co-occurrence relationship with the known category object, and the co-occurrence category information is category information that has spatial proximity, functional correlation or hierarchical relationship with the category label; a pseudo category label determination module configured to determine, based on the visual co-occurrence features and the text co-occurrence features, a pseudo category label corresponding to the co-occurrence object from the co-occurrence category information; The training module is configured to train the target detector by using the known category object and the corresponding category label, and the co-occurrence object and the corresponding pseudo-category label, so that the trained target detector is configured to output a category detection result corresponding to the image data to be processed.

9. An electronic device, comprising: A computer readable storage medium stores computer executable instructions, which, when invoked and executed by a processor, cause the processor to implement the method of any one of claims 1 to 7.

10. A computer readable storage medium characterized by The computer readable storage medium stores computer executable instructions, which, when invoked and executed by a processor, cause the processor to implement the method of any one of claims 1 to 7.

Images