Open vocabulary object detection method, apparatus, device, and storage medium

Abstract

This invention provides an open-vocabulary object detection method, apparatus, device, and storage medium, relating to the field of object detection technology. The method includes: extracting a visual feature map of the image to be detected based on an image encoder in an open-vocabulary object detection model; obtaining an alignment matrix of the visual feature map based on the positive dynamic text embedding, negative dynamic text embedding, and visual features of each target region in the visual feature map for each text category; wherein the positive and negative dynamic text embeddings for each text category are trained based on the visual features and easily confused visual features of that text category; and inputting the alignment matrix between each target region and each text category into a detection head to obtain the detection result. This invention improves the ability to distinguish easily confused categories during the detection process by establishing discriminative constraints for easily confused categories, thereby effectively improving the accuracy of open-vocabulary object detection results.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to an open vocabulary target detection method, apparatus, device, and storage medium. Background Technology

[0002] Open-vocabulary object detection models can identify any category of object that has never been seen before, even those lacking explicit bounding box labels. When detecting objects, these models construct a shared visual-semantic embedding space using a visual-language model, then map image region features and category text descriptions into the same vector space. This similarity-matching method replaces the hard-coded prediction approach using fixed classification heads found in traditional detection models.

[0003] In existing technologies, when a pre-trained open-vocabulary object detection model is applied to a specific field such as industrial quality inspection, medical imaging, aerial photography, autonomous driving street view, or underwater vision, the detection head parameters of the pre-trained open-vocabulary object detection model can be fine-tuned, and the visual encoder of the pre-trained open-vocabulary object detection model can be frozen during fine-tuning. This allows the fine-tuned open-vocabulary object detection model to achieve object detection more accurately.

[0004] However, existing open-vocabulary target detection models lack the ability to distinguish easily confused subcategories when applied, resulting in inaccurate detection results. Summary of the Invention

[0005] This invention provides an open vocabulary target detection method, apparatus, device, and storage medium to address the problem that existing open vocabulary target detection models lack the ability to distinguish easily confused subcategories, resulting in inaccurate detection results.

[0006] In a first aspect, embodiments of the present invention provide an open vocabulary target detection method, comprising: The image encoder in the open vocabulary object detection model extracts the visual feature map of the image to be detected; the visual feature map includes multiple target regions; Based on the positive dynamic text embedding, negative dynamic text embedding, visual features of each target region in the visual feature map, and geometric gating vector for each text category, the alignment matrix of the visual feature map is obtained; wherein, the positive dynamic text embedding and negative dynamic text embedding for each text category are trained based on the visual features of that text category and the easily confused visual features of that text category. The alignment matrix of the visual feature map is input into the detection head of the open vocabulary object detection model to obtain the detection result of the image to be detected.

[0007] Secondly, embodiments of the present invention provide an open vocabulary target detection device, comprising: The extraction module is used to extract visual feature maps of the image to be detected based on the image encoder in the open vocabulary object detection model; wherein, the visual feature maps include multiple target regions; The alignment module is used to obtain the alignment matrix of the visual feature map based on the positive dynamic text embedding, negative dynamic text embedding, visual features of each target region in the visual feature map, and geometric gating vector for each text category; wherein, the positive dynamic text embedding and negative dynamic text embedding for each text category are trained based on the visual features of that text category and the easily confused visual features of that text category. The detection module is used to input the alignment matrix of the visual feature map into the detection head of the open vocabulary object detection model to obtain the detection result of the image to be detected.

[0008] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect or any possible implementation thereof.

[0009] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect or any possible implementation thereof.

[0010] In this embodiment of the invention, a preset image encoder extracts the visual feature map of the image to be detected, divides the target region, and determines the corresponding visual features, providing basic visual feature support for subsequent cross-modal matching and target detection. Using positive and negative dynamic text embeddings trained based on the visual features of the text category itself and corresponding easily confused visual features, an alignment matrix between the target region and each text category can be generated by combining the visual features of the target region. This specifically strengthens the core features of the target category from the perspective of text semantic representation, while establishing discrimination constraints for easily confused categories, improving the ability to distinguish easily confused categories during the detection process. Finally, the alignment matrix is ​​input into the detection head to obtain the detection result, replacing the hard-coded prediction method of the fixed classification head in traditional detection models. While retaining the model's open vocabulary generalization ability, it significantly reduces the probability of category false detection and misclassification, effectively improving the accuracy of open vocabulary target detection results. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating the implementation of the open vocabulary target detection method provided in this embodiment of the invention; Figure 2This is a flowchart illustrating the implementation of step S120 of the open vocabulary target detection method provided in this embodiment of the invention; Figure 3 This is a flowchart illustrating the implementation of step S1202 of the open vocabulary target detection method provided in this embodiment of the invention; Figure 4 This is a schematic diagram of the open vocabulary target detection device provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0012] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0013] See Figure 1 The flowchart illustrating the implementation of the open vocabulary target detection method provided in this embodiment of the invention is described in detail below: Step S110: Extract visual feature maps of the image to be detected based on the image encoder in the open vocabulary object detection model; wherein, the visual feature maps include multiple target regions.

[0014] In some embodiments, the open vocabulary object detection model preferably adopts a CLIP-like dual-tower structure model. The model includes an image encoder and a Transformer-based text encoder, both outputting in a unified semantic embedding space. The image encoder extracts the visual representation of the input image or candidate region, i.e., a visual feature map, while the text encoder extracts the linguistic representation of the category name text, i.e., a category name embedding vector. When initializing the open vocabulary detection model, existing publicly available pre-trained weights can be directly loaded, enabling the model to have general image-text semantic alignment capabilities. Simultaneously, an open vocabulary category set is prepared, providing corresponding category name text for each text category to be detected, used to generate category name embedding vectors.

[0015] It's important to note that open vocabulary detection models learn a coarse-grained correspondence between an entire image and a whole text during pre-training, while object detection requires a precise match between a small region in an image and a specific category name. If we directly use the fixed category name embeddings output by CLIP (e.g., only the embedding vector of the word "dog") to match region features, the semantic information is too thin. It cannot distinguish the diverse appearances of the same category in different visual scenes, nor can it differentiate easily confused similar categories (such as "dog" and "wolf"). Therefore, it is necessary to divide the visual feature map into multiple regions to achieve a precise match between a small region in the image and a specific category name.

[0016] In one possible implementation, after step S110, the method further includes: dividing the visual feature map based on a pre-constructed region proposal network to obtain multiple target regions; determining the normalized area of ​​the target region as the ratio of the area of ​​each target region to the area of ​​the image to be detected; determining the aspect ratio of the target region as the ratio of the width to the height of each target region; and determining the geometric gating vector of each target region based on the normalized area and aspect ratio of each target region.

[0017] In some embodiments, after obtaining the visual feature map output by the image encoder, the visual feature map is input into the region proposal network to obtain multiple target regions. The area of ​​each target region refers to the normalized area, calculated using the following formula: , where w i h i Let W represent the width and height of the target region, and H represent the width and height of the visual feature map. The formula for calculating the aspect ratio is: After obtaining the normalized area and aspect ratio of the target region, the geometric gating vector of each target region needs to be determined according to the preset multilayer perceptron.

[0018] In one possible implementation, the specific processing method for "determining the geometric gating vector of each target region based on the normalized area and aspect ratio of each target region" is as follows: input the normalized area and aspect ratio of each target region into a preset multilayer perceptron to obtain the geometric gating vector of each target region; wherein, the dimension of the geometric gating vector is the same as the dimension of the positive dynamic text embedding and the negative dynamic text embedding of each text category.

[0019] In some embodiments, since objects of the same category exhibit different visual features at different distances—for example, small distant objects are primarily characterized by their outlines and overall shape, while large nearby objects are primarily characterized by their texture and local details—it is impossible to distinguish these scale-related differences when matching all regions using the same set of text embeddings. Therefore, a multilayer perceptron is needed to determine a geometric gating vector with the same dimensions as the positive and negative dynamic text embeddings for each text category. A pre-defined multilayer perceptron can be a two-layer MLP (Multi-Layer Perceptron). The calculation formula for a multilayer perceptron is as follows: ,in, Map 2D geometric features to a 192D hidden layer The hidden layer is mapped to 768 dimensions (the same dimension as the text embedding). , For bias.

[0020] Step S120: Based on the positive dynamic text embedding, negative dynamic text embedding, visual features of each target region in the visual feature map, and geometric gating vector of each text category, obtain the alignment matrix of the visual feature map; wherein, the positive dynamic text embedding and negative dynamic text embedding of each text category are trained based on the visual features of the text category and the easily confused visual features of the text category.

[0021] It should be noted that after obtaining multiple target regions, the RoIAlign operation can be used to extract the visual features of each target region. Where D=768 (consistent with the output dimension of CLIP ViT-L / 14). This represents a D-dimensional real vector space, where the visual features of each target region are represented as D-dimensional real vectors. For the visual features of the i-th target region, it encodes the visual appearance information of the objects within that region.

[0022] In some embodiments, the alignment matrix of the visual feature map is obtained by utilizing information such as the positive dynamic text embedding and negative dynamic text embedding of each text category, the visual features of each target region in the visual feature map, and the geometric gating vector. This allows for consideration of easily confused visual features of each text category when determining the alignment matrix, reducing the impact of easily confused features. Furthermore, the geometric gating vector of each target region can take into account the differential effects caused by image scale, thereby effectively improving the accuracy of the alignment matrix and further improving the accuracy of target detection.

[0023] See Figure 2 The specific processing method of the above step S120 includes steps S1201-S1202, and the specific content is as follows: Step S1201: Element-wise modulation of the positive and negative dynamic text embeddings for each text category is performed using the geometric gating vector of each target region in the visual feature map, to obtain the positive and negative conditional embeddings for each target region and each text category.

[0024] In some embodiments, by using the geometric gating vector of each target region in the visual feature map to perform element-wise modulation on the positive and negative dynamic text embeddings for each text category, the positive and negative conditional embeddings for each target region and each text category simultaneously encode category semantic information and region geometric information. Each target region and text category pair has its own exclusive embedding vector, which can take into account the differential impact of image scale on the positive and negative dynamic text embeddings during application. Element-wise modulation refers to the process of modulating the positive and negative dynamic text embeddings for each text category using the geometric gating vector of each target region, respectively, to obtain the exclusive positive and negative conditional embeddings for each target region-text category pair.

[0025] The formula for element-wise modulation is:

[0026]

[0027] in, It is element-wise multiplication, σ is the sigmoid function, and m i Each component is mapped to the (0,1) interval as a gating value. A gating value close to 1 preserves the semantic information of that dimension, while a value close to 0 suppresses it. When the normalized area of ​​the target region is small, the MLP learns to increase the gating value of the dimension related to "overall shape" and suppress the dimension of "local texture"; the opposite is true when the normalized area is large. The sigmoid function can ensure that the gating value is strictly within (0,1), neither completely masking nor amplifying, thus achieving flexible adjustment.

[0028] Step S1202: Based on the positive conditional embeddings and negative conditional embeddings of each target region and each text category, and the visual features of each target region, determine the alignment matrix of the visual feature map.

[0029] In some embodiments, each element in the alignment matrix of the visual feature map represents the probability that the corresponding target region belongs to a certain text category. The alignment matrix can achieve fine-grained, geometrically adaptive cross-modal alignment between visual regions and category semantics, providing accurate classification confidence for the detection head and serving as the core basis for the open vocabulary target detection classification process.

[0030] See Figure 3 The specific processing method of the above step S1202 includes steps S12021-S12022, the details of which are as follows: Step S12021: Substitute the positive conditional embedding and negative conditional embedding of each target region and each text category, and the visual features of each target region into the preset matching score calculation formula to obtain the matching score of each target region and each text category.

[0031] In some embodiments, the matching score is calculated using the following formula:

[0032] in, Positive similarity is used to measure the degree of positive semantic matching between target region i and text category c. Negative similarity is used to measure the degree of matching between "target region i and easily confused features of text category c"; It is a learnable suppression weight scalar for text category c (which can be initialized to 0.1), used to control the strength of negative repulsion, for easily confused category pairs. It will automatically increase in size to enhance its discriminative power; It is a scaling factor to prevent the high-dimensional dot product value from becoming too large, which would cause the subsequent softmax gradient to vanish. By subtracting the weighted negative from the positive, when calculating the matching score between each target region and each text category, we can not only consider what text category the target region appears to belong to, but also what text category the target region is likely to be misclassified to, thus establishing a clearer decision boundary between easily confused categories.

[0033] Step S12022: Based on the matching scores of each target region and each text category in the visual feature map, determine the alignment matrix of the visual feature map.

[0034] In some embodiments, the alignment matrix of the visual feature map is determined by the matching score of each target region and each text category in the visual feature map. This allows the alignment matrix to contain information about the degree of matching between each target region and each text category, thereby providing an accurate basis for target detection.

[0035] In one possible implementation, step S12022 is specifically processed as follows: the matching scores of each target region and each text category in the visual feature map are normalized to obtain the alignment confidence between each target region and each text category; for each alignment confidence, the number of the target region of the alignment confidence is determined as the row number of the alignment confidence, and the number of the text category of the alignment confidence is determined as the column number of the alignment confidence; based on the row number and column number of each alignment confidence, each alignment confidence is combined to obtain the alignment matrix of the visual feature map.

[0036] In some embodiments, the matching score can be normalized using softmax normalization, and the normalization calculation formula is as follows:

[0037] in, Let i be the probability that target region i belongs to text category c. For text categories The number, This represents the total number of text categories.

[0038] It should be noted that the alignment matrix of the visual feature map can be represented as:

[0039] in, Let be the probability that target region i belongs to text category c. The number of rows in the alignment matrix is ​​the number of target regions, and the number of columns in the alignment matrix is ​​the number of text categories.

[0040] In one possible implementation, the training process for the positive and negative dynamic text embeddings for each text category includes: combining a preset number of learnable vectors to obtain a positive context cue matrix and a negative context cue matrix; for each text category, concatenating the category name embedding vector of that text category with the positive and negative context cue matrices respectively to obtain the positive concatenation vector and the negative concatenation vector of that text category; supplementing the positive concatenation vector of each text category with visual attribute descriptions based on the category name of each text category, the visual features corresponding to each text category, and the fusion projection layer to obtain the initial positive dynamic text embedding for each text category; and further training the training process for the positive and negative dynamic text embeddings for each text category based on the category name of each text category, the pre-defined positive context cue matrix, and the negative context cue matrix. First, the easily confused visual features corresponding to each text category are determined, and the fusion projection layer is used to supplement the visual attribute description of the positive concatenation vector of each text category, resulting in the initial negative dynamic text embedding for each text category. Then, the open vocabulary object detection model, the initial positive dynamic text embedding for each text category, and the initial negative dynamic text embedding are used to perform object detection on the pre-labeled samples, resulting in the object detection result for each sample. The object detection result and the labeling information of each sample are compared, and the fusion projection layer parameters of the initial positive dynamic text embedding and the initial negative dynamic text embedding for each sample are optimized based on the comparison result and the pre-constructed loss function, resulting in the positive dynamic text embedding and the negative dynamic text embedding for each text category.

[0041] In some embodiments, the expressions for the positive context hint matrix and the negative context hint matrix are:

[0042]

[0043] in, each It is a learnable vector, K=16 (determined through ablation experiments: when K<8, the contextual representation ability is insufficient, and the mAP of new categories decreases by about 3%; when K>32, the parameters are redundant and the training is unstable). each It is a learnable vector.

[0044] In some embodiments, the open vocabulary category set contains N text categories. For each text category c (c=1,2,...,N), the category name text (e.g., "dog", "traffic cone") is input into the text encoder of the open vocabulary object detection model to obtain the category name embedding vector for each text category. . The linguistic semantics of the category names of text categories are encoded, but only the names themselves are included, lacking descriptive context. For example, the embedding of "dog" cannot distinguish between "a large dog standing" and "a small dog lying down." After obtaining the category name embedding vector for each text category, the category name embedding vector for each text category is concatenated with the positive context cue matrix and the negative context cue matrix, respectively, to obtain the positive concatenation vector and the negative concatenation vector for each text category. Then, the positive concatenation vector and the negative concatenation vector are fused and projected to obtain the initial positive context cue matrix and the initial negative context cue matrix.

[0045] It should be noted that, due to the simple category name Unable to distinguish visual details. By using learnable... and By concatenating and passing through a fusion projection layer (a linear layer with ReLU activation), the model can automatically supplement abstract words with specific visual attribute descriptions, such as adding features like "has fur" and "has four legs" to "dog". Furthermore, because... It is globally shared, capable of learning universal visual attribute rules across categories. Therefore, when encountering unknown new categories, it can reuse these rules to automatically supplement the simple new category name with specific visual attribute descriptions, defining the visual characteristics that the category should possess, thus achieving powerful generalization capabilities. The negative cue matrix works on the same principle, introducing... Its purpose is to identify "interference features that are prone to misjudgment". For example, "pointed snout" can be used as a negative exclusion criterion for "dog". When making identification, it can distinguish what features the category should not have, thereby establishing a clear decision boundary between easily confused categories.

[0046] The expressions for the initial positive dynamic text embedding and the initial negative dynamic text embedding are:

[0047] Concat concatenates K+1 D-dimensional vectors into a (K+1)×D=17×768=13056-dimensional vector. It is a learnable weight matrix (Xavier uniform initialization). It is a learnable bias vector (zero initialization). For initial positive dynamic text embedding, Compared to fixed It carries a rich descriptive context.

[0048]

[0049] in, and These are independent parameters of the negative projection layer. The initial negative dynamic text embedding is used to encode information about "which visual features should not belong to category c", for example, for the category "dog". They learned discriminative information such as "a pointed snout and a long face may indicate a wolf rather than a dog".

[0050] It should be noted that the training process of the initial positive dynamic text embedding and the initial negative dynamic text embedding are carried out together with the parameter fine-tuning process of the open vocabulary object detection model. When fine-tuning the open vocabulary object detection model, it is necessary to first freeze the backbone network parameters of the visual encoder of the open vocabulary object detection model, and only allow the last 1-2 layers to be fine-tuned to adapt to specific task features and retain general visual knowledge. The parameters are frozen by setting the gradient backpropagation to False. Next, after freezing the backbone network parameters of the visual encoder of the open-vocabulary object detection model, the pre-labeled samples need to be detected using the open-vocabulary object detection model, the initial positive dynamic text embeddings for each text category, and the initial negative dynamic text embeddings. The detection results for each sample include the alignment confidence of each target region and each text category, the alignment matrix of the sample, and the detection bounding box of the sample. Subsequently, the parameters of the open-vocabulary object detection model, the initial positive dynamic text embeddings, and the initial negative dynamic text embeddings are adjusted using the pre-constructed loss function and the detection results for each sample. During parameter adjustment, an end-to-end training approach can be adopted, using the Adam optimizer to update all learnable parameters, including... , Parameters of the fusion projection layer, parameters of the multilayer modulator, This includes parameters such as the region proposal network in the detection head of the open-vocabulary object detection model and the parameters of the bounding box regression layer. When adjusting the parameters, it is necessary to minimize the total loss L, and gradient updates can be achieved through backpropagation during the iteration process.

[0051] In some embodiments, the loss function includes total loss, classification loss, localization loss, and negative alignment loss; wherein the classification loss uses cross-entropy to constrain the predicted probability of the candidate region for the true class.

[0052] Where P is the set of positive sample proposals, and c* is the labeled category of the sample, which is the true category.

[0053] The localization loss can use the smooth L1 function to constrain the regression error between the detected bounding box and the ground truth bounding box.

[0054] It detects the coordinates of the bounding box. These are the coordinates of the labeled bounding box of the sample, which are also the coordinates of the true bounding box.

[0055] Negative alignment loss Its purpose is to ensure the initial negative dynamic text embedding It did indeed learn the highly confusing interference feature pattern. Specifically, it is defined as follows: for class c, the top H regions in the current training batch with the highest probability of being predicted as class c by the model but whose true class is not c are selected as the set of highly confusing negative samples. (By default, 10% of the negative samples in the batch are used). The formula for negative alignment loss is:

[0056] This loss drives the initial negative dynamic text embedding. The feature direction tends to converge towards the visual features of highly confusing negative samples, thus reducing the negative suppression term of the matching score. It can effectively reduce the classification confidence of easily confused samples.

[0057] The formula for calculating the total loss is:

[0058] in, is the positioning loss balancing weight, and μ is the negative alignment loss balancing weight (default value 0.3).

[0059] Step S130: Input the alignment matrix of the visual feature map into the detection head of the open vocabulary target detection model to obtain the detection result of the image to be detected.

[0060] In some embodiments, after obtaining the alignment matrix of the visual feature map, the alignment matrix needs to be input into the detection head of the open vocabulary object detection model. Furthermore, the matching score corresponding to each element in the alignment matrix also needs to be input into the detection head to obtain the detection result of the image to be detected. The detection head of the open vocabulary object detection model is a finely tuned detection head. Through the detection head, the alignment matrix and the matching score corresponding to each element in the alignment matrix can be used as classification confidence to participate in the calculation of detection loss and bounding box classification, thereby obtaining the detection result of the image to be detected.

[0061] It should be noted that during the incremental training phase of the open-vocabulary object detection model, positive dynamic text embedding, and negative dynamic text embedding, the system automatically monitors the prediction output, identifies high-uncertainty samples and potential error samples, and calculates the uncertainty based on the entropy value of the prediction probability.

[0062] Where x represents the input sample, c is the class index, C is the total number of classes, and p(c|x) is the model's predicted probability for class c of sample x. hour( Samples are labeled as difficult samples (with a preset threshold, e.g., 0.5); simultaneously, users report erroneous samples and samples of new categories through the annotation interface. The collected samples form an incremental dataset. Regular updates (e.g., every 30 days) and data augmentation are applied.

[0063] It should be noted that during the incremental training phase, priority should be given to updating the positive context cue matrix, the negative context cue matrix, the parameters of the fusion projection layer, the parameters of the multilayer perceptron, and the class suppression weights. The parameters of the detection head are kept frozen, with the text encoder and most of the visual encoder parameters frozen. If necessary, only a few layers at the end of the visual encoder can be selectively fine-tuned to ensure that existing general knowledge is preserved as much as possible while adapting to new categories and scenes. During training, the matching score and alignment confidence of each target region and text category are calculated, and joint optimization is performed using loss functions such as classification loss, localization loss, and negative alignment loss. This allows the dynamic alignment module and detection head to continuously adapt to new visual concepts, scene distributions, and category semantics. By iteratively optimizing the open-vocabulary object detection model, positive dynamic text embedding, and negative dynamic text embedding, new category knowledge can be continuously absorbed and existing false detection patterns can be corrected without large-scale full training, achieving dynamic expansion of open-vocabulary object detection capabilities and continuous performance evolution.

[0064] Visual feature maps of the image to be detected are extracted by a pre-set image encoder, the target region is divided and the corresponding visual features are determined, providing basic visual feature support for subsequent cross-modal matching and object detection. In addition, freezing the visual encoder backbone network can significantly reduce training overhead and retain cross-domain generalization ability. Using positive and negative dynamic text embeddings trained based on the visual features of the text category itself and the corresponding easily confused visual features, the alignment matrix between the target region and each text category can be generated by combining the visual features of the target region. This specifically strengthens the core features of the target category from the text semantic representation level, and establishes discrimination constraints for easily confused categories, improving the ability to distinguish easily confused categories during the detection process. At the same time, geometric modulation of the target region enables positive and negative conditional embeddings to adapt to targets of different scales, effectively stimulating the model's potential to recognize unseen categories and improving the detection robustness in open vocabulary scenarios. Finally, the alignment matrix is ​​input into the detection head to obtain the detection results, replacing the hard-coded prediction method of the fixed classification head in traditional detection models. While retaining the model's open vocabulary generalization ability, it significantly reduces the probability of false detection and misclassification, effectively improving the accuracy of open vocabulary object detection results.

[0065] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0066] The following are device embodiments of the present invention. For details not described in detail, please refer to the corresponding method embodiments described above.

[0067] Figure 4 A schematic diagram of the open vocabulary target detection device provided in an embodiment of the present invention is shown. For ease of explanation, only the parts related to the embodiment of the present invention are shown, and are described in detail below: like Figure 4 As shown, the open vocabulary target detection device 4 includes: Extraction module 41 is used to extract visual feature maps of the image to be detected based on the image encoder in the open vocabulary target detection model; wherein, the visual feature maps include multiple target regions; Alignment module 42 is used to obtain the alignment matrix of the visual feature map based on the positive dynamic text embedding, negative dynamic text embedding, visual features of each target region in the visual feature map, and geometric gating vector of each text category; wherein, the positive dynamic text embedding and negative dynamic text embedding of each text category are trained based on the visual features of the text category and the easily confused visual features of the text category. The detection module 43 is used to input the alignment matrix of the visual feature map into the detection head in the open vocabulary target detection model to obtain the detection result of the image to be detected.

[0068] In one possible implementation, the extraction module 41 is specifically used to: divide the visual feature map based on a pre-built region proposal network to obtain multiple target regions; determine the normalized area of ​​the target region as the ratio of the area of ​​each target region to the area of ​​the image to be detected; determine the aspect ratio of the target region as the ratio of the width to the height of each target region; and determine the geometric gating vector of each target region based on the normalized area and aspect ratio of each target region.

[0069] In one possible implementation, the extraction module 41 is further configured to: input the normalized area and aspect ratio of each target region into a preset multilayer perceptron to obtain the geometric gating vector of each target region; wherein the dimension of the geometric gating vector is the same as the dimension of the positive dynamic text embedding and the negative dynamic text embedding of each text category.

[0070] In one possible implementation, the alignment module 42 is specifically used to: modulate the positive dynamic text embedding and negative dynamic text embedding of each text category element-wise using the geometric gating vector of each target region in the visual feature map to obtain the positive conditional embedding and negative conditional embedding of each target region and each text category; and determine the alignment matrix of the visual feature map based on the positive conditional embedding and negative conditional embedding of each target region and each text category and the visual features of each target region.

[0071] In one possible implementation, the alignment module 42 is further configured to: substitute the positive conditional embedding and negative conditional embedding of each target region and each text category and the visual features of each target region into a preset matching score calculation formula to obtain the matching score of each target region and each text category; and determine the alignment matrix of the visual feature map based on the matching score of each target region and each text category in the visual feature map.

[0072] In one possible implementation, the alignment module 42 is further configured to: normalize the matching scores of each target region and each text category in the visual feature map to obtain the alignment confidence between each target region and each text category; for each alignment confidence, determine the number of the target region of the alignment confidence as the row number of the alignment confidence, and determine the number of the text category of the alignment confidence as the column number of the alignment confidence; and combine each alignment confidence according to the row number and column number of each alignment confidence to obtain the alignment matrix of the visual feature map.

[0073] In one possible implementation, the alignment module 42 is further configured to: combine a preset number of learnable vectors to obtain a positive context cue matrix and a negative context cue matrix; for each text category, concatenate the category name embedding vector of that text category with the positive context cue matrix and the negative context cue matrix respectively to obtain a positive concatenation vector and a negative concatenation vector of that text category; supplement the positive concatenation vector of each text category with visual attribute descriptions based on the category name of each text category, the visual features corresponding to each text category, and the fusion projection layer to obtain an initial positive dynamic text embedding for each text category; and based on the category name of each text category and the pre-determined text category... The corresponding easily confused visual features and the fusion projection layer supplement the visual attribute description of the positive concatenation vector of each text category to obtain the initial negative dynamic text embedding of each text category. The open vocabulary object detection model, the initial positive dynamic text embedding of each text category, and the initial negative dynamic text embedding are used to perform object detection on the pre-labeled samples to obtain the object detection result of each sample. The object detection result of each sample is compared with the annotation information, and the fusion projection layer parameters of the initial positive dynamic text embedding and the initial negative dynamic text embedding of each sample are optimized based on the comparison result and the pre-constructed loss function to obtain the positive dynamic text embedding and the negative dynamic text embedding of each text category.

[0074] Figure 5 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. For example... Figure 5 As shown, the electronic device 5 of this embodiment includes a processor 50 and a memory 51. The memory 51 stores a computer program 52. When the processor 50 executes the computer program 52, it implements the steps in the various method embodiments described above. Alternatively, when the processor 50 executes the computer program 52, it implements the functions of each module / unit in the various device embodiments described above.

[0075] For example, computer program 52 may be divided into one or more modules / units, which are stored in memory 51 and executed by processor 50 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 52 in electronic device 5.

[0076] Electronic device 5 may include, but is not limited to, processor 50 and memory 51. Those skilled in the art will understand that... Figure 5 This is merely an example of electronic device 5 and does not constitute a limitation on electronic device 5. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device 5 may also include input / output devices, network access devices, buses, etc.

[0077] The processor 50 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0078] The memory 51 can be an internal storage unit of the electronic device 5, such as a hard disk or RAM. The memory 51 can also be an external storage device of the electronic device 5, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 51 can include both internal and external storage units of the electronic device 5. The memory 51 is used to store the computer program 52 and other programs and data required by the electronic device 5. The memory 51 can also be used to temporarily store data that has been output or will be output.

[0079] For the sake of simplicity and clarity, only the above-described functional modules / units are used as examples. In practical applications, the functions described above can be assigned to different functional modules / units as needed. These modules / units can be implemented in hardware, software, or a combination of both.

[0080] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the methods described in the above-described method embodiments.

[0081] This invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the methods described in the above-described method embodiments.

[0082] Computer programs include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0083] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not detailed or described in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Unless otherwise specified or in conflict with logic, the terminology and / or descriptions between different embodiments are consistent and can be referenced interchangeably. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0084] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. An open vocabulary object detection method, characterized by, include: The image encoder in the open vocabulary object detection model extracts the visual feature map of the image to be detected; wherein, the visual feature map includes multiple target regions; The alignment matrix of the visual feature map is obtained based on the positive dynamic text embedding and negative dynamic text embedding of each text category, the visual features of each target region in the visual feature map, and the geometric gating vector; wherein, the positive dynamic text embedding and negative dynamic text embedding of each text category are trained based on the visual features of that text category and the easily confused visual features of that text category. The alignment matrix of the visual feature map is input into the detection head of the open vocabulary target detection model to obtain the detection result of the image to be detected.

2. The open vocabulary object detection method of claim 1, wherein, The step of obtaining the alignment matrix of the visual feature map based on the positive dynamic text embedding, negative dynamic text embedding of each text category, and the visual features of each target region in the visual feature map includes: The geometric gating vector of each target region in the visual feature map is used to modulate the positive dynamic text embedding and negative dynamic text embedding of each text category element by element to obtain the positive conditional embedding and negative conditional embedding of each target region and each text category. Based on the positive conditional embeddings and negative conditional embeddings of each target region and each text category, and the visual features of each target region, the alignment matrix of the visual feature map is determined.

3. The open vocabulary object detection method of claim 2, wherein, The process of determining the alignment matrix of the visual feature map based on the positive conditional embedding and negative conditional embedding of each target region and each text category, and the visual features of each target region, includes: The positive conditional embedding and negative conditional embedding of each target region and each text category, along with the visual features of each target region, are substituted into the preset matching score calculation formula to obtain the matching score for each target region and each text category. Based on the matching score of each target region and each text category in the visual feature map, the alignment matrix of the visual feature map is determined.

4. The open vocabulary object detection method of claim 3, wherein, The step of determining the alignment matrix of the visual feature map based on the matching score of each target region and each text category in the visual feature map includes: The matching scores of each target region and each text category in the visual feature map are normalized to obtain the alignment confidence between each target region and each text category; For each alignment confidence level, the number of the target region of the alignment confidence level is determined as the row number of the alignment confidence level, and the number of the text category of the alignment confidence level is determined as the column number of the alignment confidence level; Based on the row and column numbers of each alignment confidence, each alignment confidence is combined to obtain the alignment matrix of the visual feature map.

5. The open vocabulary object detection method of claim 1, wherein, The training process for the positive and negative dynamic text embeddings for each text category includes: A preset number of learnable vectors are combined to obtain a positive context cue matrix and a negative context cue matrix; For each text category, the category name embedding vector of the text category is concatenated with the positive context cue matrix and the negative context cue matrix respectively to obtain the positive concatenation vector and the negative concatenation vector of the text category; Based on the category name of each text category, the visual features corresponding to each text category, and the fusion projection layer, the visual attribute description of the forward concatenation vector of each text category is supplemented to obtain the initial forward dynamic text embedding of each text category; Based on the category name of each text category, the pre-determined easily confused visual features corresponding to each text category, and the visual attribute description supplemented by the fusion projection layer for the positive splicing vector of each text category, the initial negative dynamic text embedding of each text category is obtained; The open vocabulary target detection model, the initial positive dynamic text embedding for each text category, and the initial negative dynamic text embedding are used to perform target detection on the pre-labeled samples to obtain the target detection result for each sample. The target detection results and annotation information of each sample are compared, and the fusion projection layer parameters of the initial positive dynamic text embedding and the initial negative dynamic text embedding of each sample are optimized based on the comparison results and the pre-built loss function to obtain the positive dynamic text embedding and negative dynamic text embedding of each text category.

6. The open vocabulary object detection method of claim 1, wherein, After extracting the visual feature map of the image to be detected by the image encoder in the open vocabulary object detection model, the method further includes: The visual feature map is divided based on a pre-constructed region proposal network to obtain multiple target regions; The ratio of the area of ​​each target region to the area of ​​the image to be detected is determined as the normalized area of ​​the target region; The ratio of the width to the height of each target area is defined as the aspect ratio of that target area; The geometric gating vector for each target region is determined based on its normalized area and aspect ratio.

7. The open vocabulary object detection method of claim 6, wherein, The step of determining the geometric gating vector for each target region based on its normalized area and aspect ratio includes: The normalized area and aspect ratio of each target region are input into a preset multilayer perceptron to obtain the geometric gating vector of each target region; wherein the dimension of the geometric gating vector is the same as the dimension of the positive dynamic text embedding and the negative dynamic text embedding of each text category.

8. An open vocabulary object detection apparatus, characterized by, include: An extraction module is used to extract visual feature maps of the image to be detected based on the image encoder in the open vocabulary target detection model; wherein, the visual feature maps include multiple target regions; The alignment module is used to obtain the alignment matrix of the visual feature map based on the positive dynamic text embedding and negative dynamic text embedding of each text category, the visual features of each target region in the visual feature map, and the geometric gating vector; wherein, the positive dynamic text embedding and negative dynamic text embedding of each text category are trained based on the visual features of the text category and the easily confused visual features of the text category. The detection module is used to input the alignment matrix of the visual feature map into the detection head of the open vocabulary target detection model to obtain the detection result of the image to be detected.

9. An electronic device, comprising: It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images