Open vocabulary object detection method and system based on hierarchical semantic joint optimization

Abstract

The application discloses an open vocabulary object detection method based on hierarchical semantic joint optimization, and belongs to the cross technical field of computer vision and natural language processing. In view of the problems of limited semantic granularity, insufficient context utilization and weak generalization ability of new categories of existing methods, a structured concept dictionary is constructed, image description, category label and category definition are uniformly coded, and multi-granularity text features are obtained; image features and instance features are extracted, and neighborhood features are obtained through spatial neighborhood sampling and semantic neighborhood screening; cross-modal alignment relationships of image-level, instance-level and neighborhood-level visual features and corresponding text features are respectively established, and image-level alignment loss, instance-level contrast loss and neighborhood-level contrast loss are weighted and jointly optimized. The application can enhance multi-granularity semantic matching and the semantic association between known categories and novel categories, improve the category discrimination ability and the generalization performance in an open scene, and is suitable for general image, remote sensing image and automatic driving target recognition.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of computer vision and natural language processing, specifically to an open vocabulary target detection method and system with hierarchical semantic joint optimization. Background Technology

[0002] The closest technical approaches found mainly include: RegionCLIP's candidate region-text concept alignment, DetCLIP's category concept dictionary and definition enhancement, BARON's context region set alignment, and NRAA's neighborhood region relationship modeling. Existing technologies have focused on text concept expansion, region-level semantic matching, and neighborhood context utilization, respectively. Therefore, background technologies should avoid making a general claim that "existing technologies do not utilize definitions or context," and should instead focus on the fact that they have not yet established a definite correspondence between the three layers of text, the three layers of vision, and the differential loss.

[0003] Open-vocabulary object detection is used to identify and locate targets outside the training category set. It typically uses a vision-language pre-trained network to map the visual features of candidate regions to the text semantic space and determines the target category based on the similarity between the region features and the category text features. This type of method can expand the detection categories by utilizing the open semantics carried by the text, but the detection performance depends on the degree of adaptation between text expression, visual representation, and cross-modal alignment mechanisms.

[0004] A relatively close technical approach is to construct text features using category names, image descriptions, or category definitions, and then align candidate regions with the corresponding text. Some schemes use category names for region classification, some supplement semantics with category descriptions from a concept dictionary, and others use image descriptions to supervise region matching with words. The aforementioned texts typically perform image-level supervision, category recognition, or concept supplementation functions respectively, lacking a processing mechanism to establish a fixed hierarchical correspondence based on global scene, target instance, and fine-grained definitions. When category names are similar, target appearances are similar, or image descriptions contain multiple objects, a single text or a limited-level text cannot simultaneously define the target scene, target category, and category definition, easily leading to information mismatches between different semantic granularities.

[0005] Another approach involves supplementing contextual information beyond candidate bounding box features by utilizing neighboring regions or regions. If the neighborhood is determined solely by fixed locations or random areas, irrelevant background can easily be mixed in; if the neighborhood is selected only based on the overlap of candidate boxes, adjacent objects that are semantically related to the target but have little spatial overlap may be missed. Existing processing typically treats neighborhood information as a unified region feature for alignment, failing to clearly distinguish the textual levels corresponding to the target instance and the semantics of the neighborhood, resulting in unclear constraints of contextual information on category determination.

[0006] Furthermore, existing training methods often employ contrastive losses between regions and category text, or overall matching losses between images and text. When visual features and text features of different granularities are updated together in the same loss, their semantic scope and matching difficulty vary. Local target semantics may be affected by the global scene, and neighborhood semantics may deviate from the category definition. Therefore, how to establish matching relationships between image-level, instance-level, and neighborhood-level visual features and their corresponding text features, and maintain multi-granularity semantic consistency through mutually complementary optimization constraints, is a technical problem that needs to be solved in this field. Summary of the Invention

[0007] To address the aforementioned shortcomings of existing technologies, the present invention aims to provide an open-vocabulary object detection method based on hierarchical semantic joint optimization. This method enhances the model's ability to distinguish category semantics and generalize to new categories by accurately aligning structured text semantics with hierarchical visual features through a joint optimization learning approach.

[0008] To achieve the above objectives, this invention employs an open lexical target detection method based on hierarchical semantic joint optimization, which specifically includes the following steps:

[0009] Build a structured textual concept system;

[0010] This section aims to build a standardized and scalable structured concept dictionary, generate structured text content based on the dictionary, complete unified encoding, and form a progressive text semantic system from global and instance to fine-grained or contextual levels.

[0011] We collected image captions and other corpora, and used NLTK tools to perform word segmentation, part-of-speech tagging, and entity noun extraction. We normalized the text, extracted word roots, filtered out non-core words such as function words, adjectives, and adverbs, deduplicated nouns, and removed low-frequency noise words based on the frequency of noun occurrence (≥3). We then merged the original category labels of the dataset to obtain an initial noun list.

[0012] Furthermore, noun standardization is performed: singular and plural forms of words are standardized, uncountable nouns and fixed plural words are retained, character formats are corrected, and then, in conjunction with the WordNet knowledge base and hyponym retrieval, non-entity words such as location, scene, and abstract things are removed, and only nouns corresponding to detectable targets such as objects, food, and people are retained, resulting in a standardized noun set.

[0013] Furthermore, relying on the WordNet knowledge base, for each noun retrieval category definition within the set, the top m definitions with the highest relevance (m=5 in this invention) are selected, and a structured concept dictionary is constructed according to a fixed format. The dictionary entries are in the following form:

[0014] ,

[0015] in For the first A noun, to This is the corresponding semantic definition.

[0016] Furthermore, two types of fixed templates are used to generate label text and definition text respectively:

[0017]

[0018]

[0019] Fill the template with the nouns and their corresponding definitions in sequence to generate category label text and category definition text.

[0020] Furthermore, a frozen-state text encoder is used to encode the image captions, category label text, and category definition text respectively, obtaining the features corresponding to the three types of text:

[0021]

[0022] in, These are image captions, category label text, and category definition text, respectively. These are the corresponding text embeddings.

[0023] Constructing hierarchical visual feature representations;

[0024] This section extracts three levels of visual features—image-level, instance-level, and neighborhood-level—based on a backbone network, a region proposal network, and a fusion-based neighborhood sampling algorithm, taking into account the global layout of the image, the details of the target ontology, and the surrounding contextual information.

[0025] The backbone network extracts global features from the entire image, capturing global information such as overall spatial layout, scene atmosphere, and target distribution. Then, a region proposal network generates initial candidate boxes and extracts corresponding instance-level features, focusing on the details of the target object itself.

[0026] Furthermore, a neighborhood sampling mechanism is used to perform fusion sampling of spatial and semantic neighborhoods: multi-directional spatial neighborhoods are generated based on the target bounding box to capture spatial context. Specifically, to avoid extreme aspect ratios, the sampling probabilities in the horizontal and vertical directions are:

[0027]

[0028] Where p is the basic probability, This is the scaling factor.

[0029] Furthermore, using the current target bounding box as the center, the intersection-union ratio (IUR) between it and the other targets is calculated. If the IUR meets the threshold of 0.2 and the objectivity score of the neighboring targets meets the threshold of 0.85, it is determined that there is a strong semantic association between the two, and the neighboring target is included in the semantic neighborhood.

[0030] Furthermore, the two types of neighborhoods are merged to form a compact context region, and neighborhood-level features are extracted to enrich the surrounding association information of the target. Finally, three layers of visual features are formed: image-level, instance-level, and neighborhood-level. These features are then transformed and semantically enhanced through a semantic perception layer and encoded through a frozen text encoder to obtain the corresponding image embedding, instance embedding, and neighborhood embedding.

[0031] Joint optimization learning of semantics at different granularities;

[0032] This section establishes a one-to-one cross-modal alignment relationship at the image level, instance level, and neighborhood level, designs a hierarchical loss function and weighted fusion, and completes the model iterative optimization through joint contrastive learning to achieve semantic collaborative optimization of structured text features and hierarchical visual features.

[0033] Image-level visual features are aligned with image descriptive features to match global scene semantics. A label matrix Y is constructed to identify the correspondence between image embeddings and image caption embeddings, and bidirectional similarity between visual and textual data is calculated.

[0034]

[0035] in, This refers to the temperature parameter.

[0036] The bidirectional loss constraint is calculated using soft-label cross-entropy loss:

[0037]

[0038] Furthermore, instance-level visual features are aligned with category label features, and the Hungarian algorithm is used to achieve optimal matching between instance embeddings and category label embeddings. A bidirectional contrastive loss is constructed between the optimally matched visual and textual features:

[0039]

[0040] Furthermore, neighborhood-level visual features are aligned with category definition features. Based on the aforementioned matching index, corresponding neighborhood embeddings and category definition embeddings are extracted. Similarity is calculated, and refined category definition embeddings are selected to construct a bidirectional contrastive loss.

[0041]

[0042] Furthermore, a weighted fusion method is used to form the total loss, which collaboratively optimizes the alignment effect at different levels. This mechanism fully leverages the semantic complementarity of multiple levels to accurately uncover the potential semantic relationships between known and novel categories.

[0043] Compared with the prior art, the present invention has significant progress and unexpected technical effects:

[0044] This invention achieves multi-layered and comprehensive semantic representation capabilities, significantly enhancing the model's depth and comprehensiveness in characterizing category connotations. Compared to visual-text alignment methods that rely solely on a single form, this invention constructs a three-layered structured text description comprising image captions, category labels, and category definitions. This comprehensively covers category semantics from three dimensions: global scene, object instance, and fine-grained or contextual features. Image captions provide the overall scene context, category labels clearly identify the core category identifiers, and category definitions reveal the essential attributes of the category. These three elements complement and synergistically enhance each other, enabling the model to accurately distinguish fine-grained category differences and effectively improve the accuracy of novel category recognition.

[0045] This invention exhibits strong adaptability and stability in complex scenarios. It employs a joint sampling strategy of spatial and semantic neighborhoods to fully exploit the spatial structure and semantic relationships surrounding the target. Spatial neighborhoods supplement the spatial context, while semantic neighborhoods enhance the features of associated objects, enriching visual feature expression and improving feature recognition.

[0046] This invention establishes a hierarchical semantic joint optimization mechanism, significantly improving the accuracy and consistency of visual and text feature matching. This invention achieves three-layer cross-modal alignment: image-level, instance-level, and neighborhood-level. Image-level alignment matches global scene semantics, instance-level alignment focuses on target instance features, and neighborhood-level alignment captures contextual details. The multi-level matching and alignment work together and refine each layer, effectively solving the problems of single-level alignment's limited granularity and coarse matching, resulting in a high degree of consistency between visual features and text semantics, and significantly improving the robustness of cross-modal matching.

[0047] This invention demonstrates outstanding open-domain generalization ability, significantly improving the performance of novel category detection. It strengthens multi-level semantic association mining through joint contrastive learning at the image, instance, and neighborhood levels. The joint loss function collaboratively optimizes global, instance, and fine-grained semantic consistency, fully capturing the potential semantic connections between known and novel categories.

[0048] Compared with existing technologies, this invention does not simply process textual information, instance features, or contextual information independently in open-vocabulary object detection. Instead, it constructs a technical mechanism that coordinates structured textual concepts, hierarchical visual representations, and hierarchical semantic joint optimization, based on the correspondence between image-level, instance-level, and neighborhood-level semantics. This invention utilizes category names and their semantic definitions to form label-level and definition-level textual features, and transforms image descriptions into image-level textual features. This allows textual information of different granularities to respectively undertake global scene constraints, instance category recognition, and fine-grained semantic supplementation functions, avoiding the problems of insufficient semantic expression and limited distinguishing ability when using only category names.

[0049] In terms of visual feature processing, this invention not only extracts image-level visual features and instance-level visual features of candidate targets, but also obtains spatial and semantic neighborhoods based on the spatial relationships, intersection-over-union (IoU) ratios, and objectivity scores of candidate bounding boxes. The current candidate bounding box and the two types of neighborhoods are then merged into a single neighborhood region. This neighborhood region simultaneously contains information about the target itself, spatial context information, and semantically related information about neighboring targets. This reduces irrelevant background interference caused by fixed-range expansion and overcomes the problem of insufficient coverage of effective context based solely on candidate box overlap. The spatial and semantic neighborhoods are not simply superimposed; rather, they jointly constrain the range and content of the neighborhood region, enabling the neighborhood visual features to establish a more stable, fine-grained semantic correspondence with the category definition text.

[0050] During model training, this invention performs bidirectional soft label alignment between image-level visual semantic features and image description text features, bidirectional comparative alignment between instance-level visual semantic features and category label text features, and bidirectional comparative alignment between neighborhood-level visual semantic features and category definition text features. Then, it uses joint losses with different weights to uniformly update the trainable parameters. This hierarchical optimization method prevents a single granularity loss from dominating model training, ensuring that the global scene, local targets, and neighborhood semantics maintain mutual constraints within the same semantic space. Its technical effect is not a mechanical sum of the effects of the three losses, but rather lies in the fact that image-level semantics provides scene constraints for instance recognition, instance-level semantics ensures target category discrimination, and neighborhood-level semantics further corrects category ambiguity and missing local information, thereby improving the semantic transfer capability between basic and open categories.

[0051] Based on the aforementioned collaborative mechanism, this invention, while maintaining the text encoding network parameters unchanged, can achieve hierarchical matching of new categories with instance visual features and neighborhood visual features by updating the structured concept dictionary and encoding new category names and their definitions. This reduces the dependence of new category adaptation on retraining the base model. Especially for detection scenarios where appearances are similar but semantics differ, target local features are insufficient, targets are occluded, or there is strong interference background around the target, this invention can simultaneously utilize category names, semantic definitions, target instances, and neighborhood context for discrimination. Compared to existing solutions that only use category label alignment, fixed neighborhood expansion, or a single contrastive loss, this invention can enhance the fine-grained recognition capability of open categories while controlling neighborhood noise. This technical effect stems from the specific correspondence and joint constraint relationship between the three layers of text, three layers of vision, and three types of loss, and cannot be reasonably expected by those skilled in the art simply by combining the individual methods. Attached Figure Description

[0052] Figure 1 A flowchart illustrating the open lexical target detection method with hierarchical semantic joint optimization provided for an embodiment of the present invention;

[0053] Figure 2 A schematic diagram illustrating the process of constructing a structured concept dictionary in the hierarchical semantic joint optimization open lexical target detection method provided for an embodiment of the present invention;

[0054] Figure 3 A schematic diagram of structured text feature encoding in the hierarchical semantic joint optimization open vocabulary target detection method provided for an embodiment of the present invention;

[0055] Figure 4 A schematic diagram of hierarchical visual feature extraction in the hierarchical semantic joint optimization open vocabulary target detection method provided for an embodiment of the present invention;

[0056] Figure 5 A schematic diagram of the neighborhood sampling method in the hierarchical semantic joint optimization open vocabulary target detection method provided for an embodiment of the present invention;

[0057] Figure 6 A schematic diagram of hierarchical visual feature encoding in the open vocabulary target detection method with hierarchical semantic joint optimization provided for an embodiment of the present invention;

[0058] Figure 7 This diagram illustrates the joint optimization of hierarchical semantics in the open vocabulary target detection method with joint hierarchical semantic optimization provided for an embodiment of the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0060] This invention provides an open lexical target detection method based on hierarchical semantic joint optimization, such as... Figure 1 As shown in the accompanying drawings, the embodiments of the present invention will be described in further detail below.

[0061] Step S1: Construct a concept dictionary;

[0062] In one implementation, step S1 specifically includes the following sub-steps:

[0063] Step S11, Text corpus collection and preprocessing;

[0064] like Figure 2 As shown, we collected natural language description corpora of images (such as COCO Caption, which includes image captions, scene descriptions, etc.) and instance-level category labels. We used the NLTK natural language processing toolkit to perform word segmentation and part-of-speech tagging on all descriptive texts, and filtered out entity nouns (such as "person" and "bus") and added them to the initial noun list.

[0065] Specifically, each image description text undergoes lowercase normalization. A syntactic parsing model is used to extract core entity words and their root forms, filtering out function words, adjectives, and adverbs without actual referential meaning. Furthermore, the extracted nouns are validated using the WordNet knowledge base, retaining only semantically defined valid nouns and removing meaningless words and unrecognizable characters. Additionally, duplicate nouns appearing repeatedly in the same image description are removed to avoid duplicate counting.

[0066] Based on this, a mapping relationship between nouns and corresponding image numbers is constructed. Frequency is constrained, retaining only valid nouns that appear three or more times, while removing low-frequency noise words to ensure the validity and representativeness of the nouns, thus forming an initial noun list. The original category labels and candidate concepts in the dataset are then merged to finally form the initial noun list.

[0067] Step S12, noun standardization processing;

[0068] During the merging of the noun list, standardization cleaning is carried out simultaneously. Specifically, this includes unifying singular and plural variants (e.g., unifying the plural form of "dogs" to the singular "dog"), retaining uncountable nouns (e.g., "water" and "bread"), and fixing plural forms (e.g., "children" and "sheep").

[0069] Furthermore, the merged nouns undergo symbol standardization, eliminating redundant separators and invalid spaces. For example, "air plane" is corrected to "airplane" to ensure that the noun format is consistent and standardized.

[0070] Furthermore, through hypernym and hyponym retrieval and semantic hierarchy judgment, non-entity target words such as location, scene, abstract things, images themselves, and structural categories are removed, and only valid nouns belonging to detectable target categories such as objects, food, and people are retained, resulting in a standardized noun set that is non-redundant, uniform in format, and semantically valid. .

[0071] Step S13, knowledge retrieval and dictionary construction;

[0072] Using WordNet as an external knowledge source, a standardized set of nouns was analyzed. Each noun in Semantic information is retrieved item by item. For each noun, the first noun with the highest semantic relevance to it is retained. Definition (in this invention) (This can be adjusted according to actual needs). Each definition must accurately reflect the core semantics of the noun in different scenarios. The nouns and their corresponding definitions are organized into a structured concept dictionary according to a preset format. The dictionary entries are organized as follows:

[0073] ,

[0074] in For the first A noun, to To define the corresponding semantics, we can enhance and expand the semantics of the categories.

[0075] Step S2, annotation data collection and preprocessing;

[0076] In one implementation, step S2 specifically includes the following sub-steps:

[0077] Step S21, annotation data collection;

[0078] Iterate through all image numbers, sequentially reading the original image information and all corresponding annotation information for each image. The image information includes basic parameters such as image file name, size, and image number, while the annotation information includes image captions.

[0079] Step S22: Annotated data preprocessing and category definition retrieval;

[0080] The image captions are parsed to obtain a list of category labels. Each category label in the list is iterated over, and its corresponding category definition is retrieved from a concept dictionary. The category labels and their definitions are stored in a category definition list.

[0081] The raw image information and annotation information of each image are structured and parsed to generate standardized data units required for model training. During the parsing process, the text annotations, category labels, and category definition information corresponding to the image are extracted to form image-level, label-level, and definition-level text data structures, resulting in structured text information, which is called structured text concepts.

[0082] Step S3, structured text concept encoding;

[0083] In one implementation, step S3 specifically includes the following sub-steps:

[0084] Step S31, category label preprocessing;

[0085] Populate a predefined template with category labels

[0086]

[0087] Each noun in the category definition list Fill into the template Location-based, generating concise category label text (such as "A photo of cat", "A photo of chair"), focusing on the core semantics of the category name.

[0088] Step S22, category definition preprocessing;

[0089] Populate the predefined template using category definitions

[0090]

[0091] Each noun in the category definition list The template is populated with its five corresponding definitions to generate five category definition texts (such as "This is a photo of a cat, which is a small domesticated carnivorous mammal"), supplementing the detailed text information of the category concept.

[0092] Step S33, Image caption filtering;

[0093] Select the original natural language description corresponding to the input image (such as the 5 captions for each image in the COCO dataset) as the image-level text description, and preserve the scene context information of the target object.

[0094] Step S34, Structured text concept encoding;

[0095] like Figure 3 As shown, the ViT-B / 32 version of CLIP is used as the text encoder. During training, the encoder parameters are kept frozen to ensure the stability of the semantic space. The image captions, category label text, and category definition text are respectively input into the text encoder, and the corresponding text features are output through encoding operations. (Image caption embedding) (Category tag embedding) (Category definition embedding) All text is mapped to a unified semantic space.

[0096] Step S4: Extract image features and instance features;

[0097] In one implementation, such as Figure 4 As shown:

[0098] A ResNet-50 backbone network is used to extract features from the input image and output image features. The feature maps are then input into a Region Proposal Network (RPN) to generate target candidate boxes. Non-Maximum Suppression (NMS) is performed on the initial candidate boxes to obtain a deredundant list of target objects.

[0099] Step S5: Calculate and extract neighborhood features using a neighborhood sampling algorithm;

[0100] In one implementation, such as Figure 5 As shown, step S5 includes the following sub-steps:

[0101] Step S51, spatial neighborhood expansion;

[0102] With each target object Candidate box size ( Based on the above, an anchor frame is constructed. Four basic neighborhood frames are generated by sliding along four orthogonal directions: top, bottom, left, and right. Then, using the aspect ratio of these four neighborhood frames as a reference, four supplementary neighborhood frames are generated by sliding along four diagonal directions: top left, top right, bottom left, and bottom right. A total of eight initial spatial neighborhood candidate regions are obtained. Cropping of neighborhood boxes that extend beyond the image boundaries avoids interference from invalid regions; setting the base sampling probability. scaling factor Calculate the horizontal sampling probability:

[0103] ,

[0104] Vertical sampling probability:

[0105] ,

[0106] The sampling probability of each candidate region is dynamically adjusted to prevent extreme aspect ratios.

[0107] Step S52: Semantic neighborhood filtering;

[0108] With target object Centered on the target object, calculate its relationship with other target objects in the list. Crossover ratio ( );set up threshold Objectivity score threshold ,like satisfy And the objectivity score Then determine and There is a strong semantic interaction relationship, so it is included. Semantic neighborhood candidate regions.

[0109] Step S53: Spatial neighborhood and semantic neighborhood are fused;

[0110] Merge candidate regions from the spatial neighborhood (GN) and semantic neighborhood (SN), and calculate the minimum bounding box covering all candidate regions as the minimum bounding box. The final context neighborhood representation, and the coordinate fusion calculation rules are as follows:

[0111]

[0112] This neighborhood includes both the target object itself and its related context. Features of the corresponding region, known as neighborhood features, are extracted from the image feature map based on the neighborhood coordinates.

[0113] Step S6, hierarchical visual feature encoding;

[0114] In one implementation, such as Figure 6 As shown:

[0115] Image-level, neighborhood-level, and instance-level visual features are fused together, and their corresponding positional codes are preserved to retain spatial structure information. These are then input into a two-layer convolutional semantic perception layer (Vision to Language, V2L) for feature dimensionality transformation and semantic enhancement. The fused features are then input into a frozen CLIP text encoder, mapping them to the same semantic space as the text features to obtain the visual mapping embedding. (Image-level mapping embedding) (Neighborhood-level mapping embedding) (Instance-level mapping embedding).

[0116] Step S7: Joint optimization of semantics at different granularities;

[0117] In one implementation, such as Figure 7 As shown, step S7 includes the following sub-steps:

[0118] Step S71, Image-level semantic alignment;

[0119] Set temperature parameters Construct a soft tag matrix If candidate region With description From the same image, then ,otherwise ;calculate and soft label cross-entropy loss :

[0120]

[0121]

[0122] Promote consistency between global visual features and image caption semantics.

[0123] Step S72, instance-level matching;

[0124] Using the Hungarian algorithm (Instance-level mapping embedding) and (Category label embedding) Perform optimal matching to obtain the visual matching index. Language-side matching index Extract paired samples based on the index. ( (Matching samples) ( (Matching samples), constructing instance-level positive sample pairs.

[0125] Step S73, instance-level alignment;

[0126] Calculate instance-level contrastive learning loss :

[0127]

[0128] Here, T is the set of category label text embeddings, and V is the set of instance embeddings. In addition to the category label text embeddings and instance embeddings of the current training batch, there are global text variables and global visual variables used to continuously store the category label text embeddings and instance embeddings of adjacent training batches, thereby enhancing the effect of contrastive learning. Loss constraints can strengthen the semantic association between the target object and the category label.

[0129] Step S74, neighborhood-level matching;

[0130] Based on visual matching index Language-side matching index ,exist (Neighborhood-level mapping embedding) and Extract the corresponding paired features from the category definition embedding. Then, calculate the cosine similarity between the neighborhood embedding in the paired features and each definition in the category definition embedding, retaining the definition with the highest similarity to obtain the refined text embedding. Extracting paired samples ( (medium sample) (Refine the definition of embedding) to construct neighborhood-level positive sample pairs.

[0131] Step S75, neighborhood-level alignment;

[0132]

[0133] Here, T is the set of category definition text embeddings, and V is the set of neighborhood embeddings. In addition to the category definition text embeddings and neighborhood embeddings of the current training batch, there are global variables used to continuously store the category definition text embeddings and neighborhood embeddings of adjacent training batches, thereby enhancing the effect of contrastive learning. Loss constraints can improve the consistency between the target contextual features and the detailed semantic representation.

[0134] Step S76, joint optimization of hierarchical semantics;

[0135] Set the total loss function:

[0136]

[0137] in , , To balance the contribution weights of semantic learning at each level.

[0138] To verify the effectiveness and superiority of the open vocabulary target detection method based on hierarchical semantic joint optimization proposed in this invention, experimental verification was conducted. The performance evaluation index was the mean accuracy of the new category (IoU=0.5, denoted as AP). n 50 Furthermore, there is the known class mean precision (IoU=0.5, denoted as AP). b 50 ) and the mean precision of all classes (IoU=0.5, denoted as AP) all 50 (For reference only.) The training dataset of this invention is COCO 2017, which contains 118,287 training images and 5,000 validation images.

[0139] For simplicity, this invention uses HSJO (Open-Vocabulary Object Detection with Hierarchical Semantic Joint Optimization) to represent the proposed open-vocabulary object detection method based on hierarchical semantic joint optimization. A comparison of the average precision of the new categories is shown in Table 1.

[0140] In order to express simplicity, compare the methods F-VLM (F-VLM: Open-Vocabulary Object DetectionUpon Frozen Vision and Language Models), BIRDet (Boosting Open-VocabularyObject Detection by Handling Background Samples), DCS (Open-vocabulary object detection via debiased curriculum self-training), VLDet (Learning Object-Language Alignments for Open-Vocabulary Object) Detection), CLIM (CLIM: Contrastive Language-Image Mosaic for Region Representation), CondHead (Learning to Detect and Segment for Open Vocabulary Object Detection), DST-Det (DST-Det: Open-Vocabulary Object Detection via Dynamic Self-Training), CODet (Benefit From Seen: Enhancing Open-Vocabulary Object Detection by BridgingVisual and Textual Co-Occurrence Knowledge)、Baron(Aligning Bag of Regions for Open-Vocabulary Object Detection), CoDet (CoDet: Co-Occurrence Guided Region-Word Alignment for Open-Vocabulary Object Detection), DVDet (LLMS MEET VLMS: Boost Open Vocabulary Object with Fine-Grained Descriptors) are represented by abbreviations.

[0141] Table 1. Comparison of mean accuracy of new classes for different methods in open vocabulary object detection task.

[0142]

[0143] Where "-" indicates that the result is not explicitly labeled, AP n 50 As the primary reference item, AP b 50 and AP all 50 This is a secondary reference. The method proposed in this invention shows significant advantages over novel class learning and detection methods that are not explicitly labeled during training.

[0144] Furthermore, a comparison of the preprocessing methods for different text formats before encoding is shown in Table 2.

[0145] Table 2. Preprocessing methods for text forms before encoding in different methods for open-vocabulary object detection tasks.

[0146]

[0147] In this table, "-" indicates no explicit labeling. Preprocessing techniques for different text formats are key technical aspects, as shown in Table 2. The method proposed in this invention uses targeted predefined templates for category labels and category definitions, and the category definitions are obtained from a reusable and scalable concept dictionary. First, concept dictionary retrieval is particularly important in the semantic matching and alignment process between category definitions and instance neighborhoods. Rich category definition calculation and selection ensure consistency between visual and textual semantic expression. Second, the instance neighborhoods matched and aligned with the category definitions are calculated and extracted by a neighborhood sampling algorithm. Instance neighborhoods that integrate spatial structure and semantic association contain more accurate fine-grained information or scene context information in terms of semantic scope. Third, the alignment and joint optimization of semantic representation scope—including textual semantic alignment between the overall image and the global scene, alignment of visual information of the target ontology with the core semantics of the category name, and alignment of the target neighborhood with fine-grained or contextual semantics—achieves hierarchical accurate matching and collaborative learning, fully releasing the value of multi-granular semantics.

[0148] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A training method for an open lexical target detection model based on hierarchical semantic joint optimization, characterized in that, include: The image description text, category label text, and category definition text corresponding to the training image are obtained. The image-level text features, label-level text features, and definition-level text features are obtained by keeping the input parameters of the image description text, category label text, and category definition text unchanged. The training images are input into the backbone network to obtain image-level visual features. Candidate bounding boxes are generated based on the image-level visual features, and instance-level visual features corresponding to each candidate bounding box are extracted. For each candidate bounding box, a spatial neighborhood is determined based on the spatial range of the candidate bounding box, and a semantic neighborhood is determined based on the intersection-union ratio and objectivity score between the candidate bounding boxes. The current candidate bounding box, the spatial neighborhood, and the semantic neighborhood are merged into a neighborhood region, and the neighborhood-level visual features corresponding to the neighborhood region are extracted. The image-level visual features, instance-level visual features, and neighborhood-level visual features are respectively input into the vision-to-language mapping network to obtain image-level visual semantic sequences, instance-level visual semantic sequences, and neighborhood-level visual semantic sequences that are adapted to the input format of the text encoding network. Each visual semantic sequence is then input into the text encoding network to obtain image-level visual semantic features, instance-level visual semantic features, and neighborhood-level visual semantic features, respectively. Image-level soft label cross-entropy loss is obtained based on the bidirectional similarity between image-level visual semantic features and image-level text features; instance-level bidirectional contrast loss is obtained based on the matching instance-level visual semantic features and label-level text features; and neighborhood-level bidirectional contrast loss is obtained based on the matching neighborhood-level visual semantic features and definition-level text features. The image-level soft-label cross-entropy loss, instance-level bidirectional contrast loss, and neighborhood-level bidirectional contrast loss are weighted and summed according to preset weights to obtain a joint loss. The trainable parameters of the open vocabulary object detection model are updated according to the joint loss, while keeping the parameters of the text encoding network unchanged.

2. The training method for an open lexical target detection model based on hierarchical semantic joint optimization according to claim 1, characterized in that, The category label text and category definition text are generated through a structured concept dictionary, the construction of which includes: Collect publicly available image description corpora and instance category tags, and perform word segmentation, part-of-speech tagging, and noun extraction on the publicly available image description corpora and instance category tags to obtain an initial noun list; The initial noun list is subjected to case unification, singular and plural normalization, and format correction. Invalid words are filtered out, and abstract nouns and non-entity nouns related to scenarios are removed to form a standardized noun set. For each standardized noun in the standardized noun set, the corresponding semantic definition is retrieved from an external lexical semantic knowledge base, and 1 to 5 core definitions are retained in the order of retrieval. Establish the association between the standardized nouns and their corresponding core definitions, so that each dictionary entry of the structured concept dictionary includes a standardized noun field and a semantic definition field.

3. The training method for an open lexical target detection model based on hierarchical semantic joint optimization according to claim 2, characterized in that: The original image description corresponding to the training image is determined as the image description text; The category label text is generated according to the text relationship of "This is a photo of class name", where class name is replaced by the standardized noun in the structured concept dictionary; The category definition text is generated according to the text relationship "This is a photo of class name, which is definition", where the class name is replaced by the standardized noun and the definition is replaced by the semantic definition corresponding to the standardized noun. The image description text, category label text, and category definition text are encoded into the same semantic space.

4. The training method for an open lexical target detection model based on hierarchical semantic joint optimization according to claim 1, characterized in that, The determination of the spatial neighborhood and semantic neighborhood includes: Set 8 initial regions in the neighborhood space of the current candidate bounding box; The horizontal sampling probability is determined based on the ratio of the height to the width of the current candidate bounding box. First, the ratio of height to width is multiplied by a scaling factor, and the smaller value between the resulting product and 1 is taken. Then, the smaller value is multiplied by the base probability. The vertical sampling probability is determined based on the ratio of the width to the height of the current candidate bounding box. First, the ratio of the width to the height is multiplied by the scaling factor, and the smaller value between the resulting product and 1 is taken. Then, the smaller value is multiplied by the base probability. The spatial neighborhood is obtained from the eight initial regions according to the horizontal and vertical sampling probabilities, wherein the base probability is 0.5 and the scaling factor is 3.

0. The semantic neighborhood is obtained by selecting candidate bounding boxes from the neighboring candidate bounding boxes of the current candidate bounding box that have an intersection-union ratio greater than 0.2 and an object property score greater than 0.

85. The minimum bounding rectangle of the current candidate bounding box, the spatial neighborhood, and the semantic neighborhood is determined as the neighborhood region. Instance-level visual features corresponding to the current candidate bounding box and neighborhood-level visual features corresponding to the neighborhood region are extracted from the image-level visual features, respectively.

5. The training method for an open lexical target detection model based on hierarchical semantic joint optimization according to claim 1, characterized in that, The acquisition of the joint loss includes: The similarity between the image-level visual semantic features and the image-level text features is calculated separately. The similarity between the two directions is scaled using a temperature coefficient. The soft label cross-entropy loss in the two directions is obtained according to the soft label matrix. The arithmetic mean of the soft label cross-entropy loss in the two directions is determined as the image-level soft label cross-entropy loss. Matching instance-level visual semantic features and label-level text features are identified as instance-level positive sample pairs, and unmatched instance-level visual semantic features and label-level text features in the same sample batch are identified as instance-level negative sample pairs. Based on the normalization relationship between positive sample similarity scaled by temperature coefficient and negative sample similarity, instance-level losses in the visual-to-text direction and text-to-visual direction are obtained respectively, and the arithmetic mean of the instance-level losses in the two directions is determined as the instance-level bidirectional contrast loss. The neighborhood-level visual semantic features and definition-level text features that match and have the highest semantic relevance are identified as neighborhood-level positive sample pairs. The neighborhood-level visual semantic features and definition-level text features that do not match in the same sample batch are identified as neighborhood-level negative sample pairs. Based on the normalization relationship between the similarity of positive samples scaled by the temperature coefficient and the similarity of negative samples, the neighborhood-level loss in the visual-to-text direction and the text-to-visual direction are obtained respectively. The arithmetic mean of the neighborhood-level loss in the two directions is determined as the neighborhood-level bidirectional contrast loss. The weight of the image-level soft-label cross-entropy loss is set to 1, the weight of the instance-level bidirectional contrast loss is set to 0.5, and the weight of the neighborhood-level bidirectional contrast loss is set to 0.

25.

6. An open lexical target detection method based on hierarchical semantic matching, characterized in that, include: The process involves acquiring the image to be detected and the category to be detected, generating category label text and category definition text based on the category name and category definition of each category, and using a text encoding network with unchanged input parameters for the category label text and category definition text to obtain the label-level text features and definition-level text features corresponding to each category to be detected. The image to be detected is input into an open vocabulary object detection model to extract image-level visual features. Candidate bounding boxes are generated based on the image-level visual features, and instance-level visual features corresponding to each candidate bounding box are extracted. For each candidate bounding box, a spatial neighborhood is determined based on the spatial range of the candidate bounding box, and a semantic neighborhood is determined based on the intersection-union ratio and objectivity score between the candidate bounding boxes. The current candidate bounding box, the spatial neighborhood, and the semantic neighborhood are merged into a neighborhood region, and the neighborhood-level visual features corresponding to the neighborhood region are extracted. The instance-level visual features and neighborhood-level visual features are respectively input into the vision-to-language mapping network to obtain instance-level visual semantic sequences and neighborhood-level visual semantic sequences that are adapted to the input format of the text encoding network. The instance-level visual semantic sequences and neighborhood-level visual semantic sequences are then input into the text encoding network to obtain instance-level visual semantic features and neighborhood-level visual semantic features. Calculate the instance-level similarity between the instance-level visual semantic features and each label-level text feature, calculate the neighborhood-level similarity between the neighborhood-level visual semantic features and each definition-level text feature, determine the detection category corresponding to each candidate bounding box based on the instance-level similarity and neighborhood-level similarity, and output the target bounding box and the corresponding category.

7. The open lexical target detection method based on hierarchical semantic matching according to claim 6, characterized in that, The category label text and category definition text are generated through a structured concept dictionary, where each dictionary entry includes interrelated standardized nouns and 1 to 5 semantic definitions; Generate the category label text based on the standardized nouns, and generate the category definition text based on the standardized nouns and their corresponding semantic definitions; The category label text is used to match instance-level visual semantic features, and the category definition text is used to match neighborhood-level visual semantic features.

8. The open vocabulary target detection method based on hierarchical semantic matching according to claim 6, characterized in that, The acquisition of the neighborhood region includes: Eight initial regions are set in the neighborhood space of the current candidate bounding box. The horizontal sampling probability is determined according to the ratio of the height to the width of the current candidate bounding box, and the vertical sampling probability is determined according to the ratio of the width to the height of the current candidate bounding box. The spatial neighborhood is obtained according to the horizontal and vertical sampling probabilities. The horizontal sampling probability is determined according to the following relationship: first, multiply the ratio of height to width by 3.0, take the smaller value between the resulting product and 1, and then multiply the smaller value by 0.5; The vertical sampling probability is determined according to the following relationship: first, multiply the ratio of width to height by 3.0, take the smaller value between the resulting product and 1, and then multiply the smaller value by 0.5; Select neighboring candidate bounding boxes with an intersection-union ratio greater than 0.2 and an object property score greater than 0.85 to obtain semantic neighborhoods; The smallest bounding rectangle covering the current candidate bounding box, the spatial neighborhood, and the semantic neighborhood is determined as the neighborhood region.

9. The open lexical target detection method based on hierarchical semantic matching according to claim 6, characterized in that, Determining the detection category corresponding to each candidate bounding box includes: For each candidate bounding box, the instance-level similarity between the instance-level visual semantic features and the label-level text features of each category to be detected is calculated to form an instance-level category matching result; Calculate the neighborhood-level similarity between the neighborhood-level visual semantic features and the definition-level text features of each category to be detected, and form a neighborhood-level category matching result; The instance-level category matching results and the neighborhood-level category matching results are associated according to the correspondence of the categories to be detected, and the category to be detected corresponding to the current candidate bounding box is determined based on the associated instance-level similarity and neighborhood-level similarity.

10. The open vocabulary target detection method based on hierarchical semantic matching according to claim 7, characterized in that: Standardized nouns and their semantic definitions are added to the structured concept dictionary according to the detection scenario; Generate new category label text based on the newly added standardized nouns, and generate new category definition text based on the newly added standardized nouns and their semantic definitions; The newly added category label text and the newly added category definition text are input into the text encoding network to obtain the label-level text features and definition-level text features corresponding to the newly added category. The label-level text features corresponding to the newly added category are matched with the instance-level visual semantic features, and the definition-level text features corresponding to the newly added category are matched with the neighborhood-level visual semantic features. The target detection results of the newly added category are output while keeping the parameters of the open vocabulary target detection model unchanged.

Images