An open vocabulary object detection method and system

Abstract

The application provides an open vocabulary target detection method and system, which comprises the following steps: integrating a learnable prompt anchor module in a multi-scale feature layer of a preset cross-modal feature fusion network; injecting each prompt anchor point vector into a corresponding feature map through residual connection to dynamically correct the multi-scale semantic bias of the feature map; in the model training process, using a frozen CLIP visual encoder as a teacher model and the preset cross-modal feature fusion network as a student model to train a corresponding small target region mask, and calculating the distillation loss of the dynamically corrected feature map according to the small target region mask; outputting the multi-scale features corresponding to the feature map according to the distillation loss, and performing boundary box regression and open vocabulary classification on the multi-scale features through a preset detection head to complete the corresponding target detection. The application can effectively improve the detection efficiency of small region targets.

Description

An Open Vocabulary Target Detection Method and System Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to an open vocabulary target detection method and system. Background Technology

[0002] Open vocabulary object detection is an important research direction in the field of computer vision. Its goal is to enable models to identify categories that have not appeared in the training set, thereby greatly enhancing the model's generalization ability and practical value.

[0003] Among them, with the continuous development of technology, existing technologies have developed representative methods such as CLIP (contrastive language pre-trained model), GLIP (visual language pre-trained model), and YOLO-World (open vocabulary object detection model), which can be applied to different scenarios.

[0004] Furthermore, in practical applications, existing technologies often rely on large-scale data in training data to dominate text embedding for broad categories such as "people" and "vehicles." This results in low confidence for small target instances during visual-linguistic alignment, making them prone to missed detection. Additionally, during training, the gradients of large-scale targets can overwrite the gradients of small targets, making it difficult for the model to learn the strong discriminative features of small targets. Consequently, it becomes difficult to effectively focus on small target regions, thus reducing detection efficiency. Summary of the Invention

[0005] Based on this, the purpose of the present invention is to provide an open vocabulary target detection method and system to solve the problem that the existing technology is difficult to effectively focus on small target areas, which leads to a reduction in detection efficiency.

[0006] The first aspect of the present invention proposes:

[0007] An open-vocabulary target detection method specifically includes the following steps:

[0008] In the multi-scale feature layer of the preset cross-modal feature fusion network, a learnable cue anchor module is integrated. The learnable cue anchor module is used to define a set of learnable cue anchor vectors for each feature map layer.

[0009] Each of the aforementioned cue anchor vectors is injected into its corresponding feature map via residual connections to dynamically correct the multi-scale semantic bias of the feature map.

[0010] During model training, the frozen CLIP visual encoder is used as the teacher model and the preset cross-modal feature fusion network is used as the student model to train the corresponding small target region mask, and the distillation loss of the dynamically corrected feature map is calculated based on the small target region mask.

[0011] The multi-scale features corresponding to the feature map are output based on the distillation loss, and the multi-scale features are then subjected to bounding box regression and open vocabulary classification by a preset detection head to complete the corresponding target detection.

[0012] The beneficial effects of this invention are as follows: by combining the feature map with the cue anchor vector, the effective features contained within the feature map can be extracted more accurately. Based on this, in order to further improve the accuracy of subsequent training, the current feature map will be dynamically corrected. Based on this, during the model training process, teacher models and student models can be set separately, and multi-scale features of the feature map can be obtained. Based on this, the current multi-scale features are finally used for bounding box regression and open vocabulary classification, thereby accurately completing the corresponding target detection, and effectively focusing on small target regions, thus improving detection efficiency.

[0013] Furthermore, the step of injecting each of the cue anchor vectors into its corresponding feature map via residual connections to dynamically correct the multi-scale semantic bias of the feature map includes:

[0014] Once the feature map is determined, a cue anchor vector corresponding to the feature map is defined;

[0015] The original vector size of the prompt anchor vector is expanded to the same target vector size as the feature map size through a broadcast operation, and the corrected feature map is obtained by adding the residuals.

[0016] Furthermore, the step of training the corresponding small target region mask includes:

[0017] During model training, corresponding annotation information is added to the feature map;

[0018] Based on the annotation information, an initial region with an area smaller than a preset area threshold and an anchor frame intersection-union ratio greater than a preset intersection-union ratio threshold is selected from within the feature map;

[0019] The initial region is set as the small target region, and a corresponding binary mask is generated to set the binary mask as the small target region mask.

[0020] Furthermore, the step of calculating the distillation loss of the dynamically corrected feature map based on the small target region mask includes:

[0021] The first feature output by the student model is mapped to the same semantic space as the second feature output by the teacher model using a linear projection function.

[0022] Based on the semantic space, the distillation loss is calculated using a preset algorithm according to the first feature and the second feature.

[0023] Furthermore, the expression for the preset algorithm is:

[0024]

[0025] Among them, F i (s) For student model features, F i (t) For teacher model features, M is the numerically stable term. i This represents the mask for the small target region.

[0026] Furthermore, the step of performing bounding box regression and open vocabulary classification on the multi-scale features using a preset detection head to complete the corresponding target detection includes:

[0027] The category names or natural language prompts of the multi-scale features are converted into corresponding text embeddings;

[0028] Visual features corresponding to the feature map are extracted from the multi-scale features, and the feature map is classified based on the text embedding and the visual features.

[0029] Furthermore, the step of classifying the feature map based on the text embedding and the visual features includes:

[0030] The cosine similarity between the text embedding and the visual features is calculated, and the feature map is classified based on the magnitude of the cosine similarity.

[0031] The second aspect of the present invention proposes:

[0032] An open vocabulary target detection system, wherein the system comprises:

[0033] An integration module is used to integrate a learnable cue anchor module into the multi-scale feature layers of a preset cross-modal feature fusion network. The learnable cue anchor module is used to define a set of learnable cue anchor vectors for each feature map layer.

[0034] The correction module is used to inject each of the cue anchor vectors into its corresponding feature map through residual connections, so as to dynamically correct the multi-scale semantic bias of the feature map.

[0035] The training module is used to train the corresponding small target region mask by using the frozen CLIP visual encoder as the teacher model and the preset cross-modal feature fusion network as the student model during the model training process, and to calculate the distillation loss of the dynamically corrected feature map based on the small target region mask.

[0036] The classification module is used to output multi-scale features corresponding to the feature map based on the distillation loss, and to perform bounding box regression and open vocabulary classification on the multi-scale features through a preset detection head to complete the corresponding target detection.

[0037] Furthermore, the correction module is specifically used for:

[0038] Once the feature map is determined, a cue anchor vector corresponding to the feature map is defined;

[0039] The original vector size of the prompt anchor vector is expanded to the same target vector size as the feature map size through a broadcast operation, and the corrected feature map is obtained by adding the residuals.

[0040] Furthermore, the training module is specifically used for:

[0041] During model training, corresponding annotation information is added to the feature map;

[0042] Based on the annotation information, an initial region with an area smaller than a preset area threshold and an anchor frame intersection-union ratio greater than a preset intersection-union ratio threshold is selected from within the feature map;

[0043] The initial region is set as the small target region, and a corresponding binary mask is generated to set the binary mask as the small target region mask.

[0044] Furthermore, the training module is specifically used for:

[0045] The first feature output by the student model is mapped to the same semantic space as the second feature output by the teacher model using a linear projection function.

[0046] Based on the semantic space, the distillation loss is calculated using a preset algorithm according to the first feature and the second feature.

[0047] Furthermore, the expression for the preset algorithm is:

[0048]

[0049] Among them, F i (s) For student model features, F i(t) For teacher model features, M is the numerically stable term. i This represents the mask for the small target region.

[0050] Furthermore, the classification module is specifically used for:

[0051] The category names or natural language prompts of the multi-scale features are converted into corresponding text embeddings;

[0052] Visual features corresponding to the feature map are extracted from the multi-scale features, and the feature map is classified based on the text embedding and the visual features.

[0053] Furthermore, the classification module is specifically used for:

[0054] The cosine similarity between the text embedding and the visual features is calculated, and the feature map is classified based on the magnitude of the cosine similarity.

[0055] The third aspect of the present invention proposes:

[0056] A computer includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the open vocabulary target detection method as described above.

[0057] The fourth aspect of the present invention proposes:

[0058] A readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the open vocabulary target detection method as described above.

[0059] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0060] Figure 1 is a flowchart of the open vocabulary target detection method provided in the first embodiment of the present invention;

[0061] Figure 2 is a structural block diagram of the open vocabulary target detection system provided in the third embodiment of the present invention.

[0062] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0063] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0064] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0066] Please refer to Figure 1, which shows the open vocabulary target detection method provided in the first embodiment of the present invention. The open vocabulary target detection method provided in this embodiment can accurately and effectively detect features in small target regions, thereby effectively completing the detection of open vocabulary and improving the detection efficiency.

[0067] Specifically, this embodiment provides:

[0068] An open-vocabulary target detection method specifically includes the following steps:

[0069] Step S10: In the multi-scale feature layer of the preset cross-modal feature fusion network, a learnable cue anchor module is integrated. The learnable cue anchor module is used to define a set of learnable cue anchor vectors for each feature map layer.

[0070] It should be noted that, firstly, a learnable cue anchor module is integrated into the multi-scale feature layer of the pre-defined cross-modal feature fusion network. Specifically, the cross-modal feature fusion network is used to fuse visual features (image information) and text features (category vocabulary), while the multi-scale feature layer can extract image features at different resolutions (e.g., low-resolution features of large objects and high-resolution features of small objects). The learnable cue anchor module defines a set of learnable cue anchor vectors for each feature map layer. These vectors capture key semantic information at that scale (e.g., the overall contour semantics of "vehicle" at a low scale and the detailed semantics of "wheel" at a high scale), providing a semantic benchmark for subsequent feature correction to facilitate subsequent processing.

[0071] Step S20: Each of the aforementioned cue anchor vectors is injected into its corresponding feature map through residual connection in order to dynamically correct the multi-scale semantic bias of the feature map.

[0072] It's worth noting that, next, the cue anchor vector is injected into the corresponding feature map via residual connections. Residual connections can introduce semantic biases of the cue anchors (such as emphasizing the contour features of "pedestrian") while preserving the original feature information, achieving dynamic correction of multi-scale semantic biases. Specifically, for example, low-scale feature maps may focus more on global semantics, while high-scale maps focus more on local details. After correction, the semantic consistency of features at different scales can be ensured, avoiding category confusion caused by scale differences. This facilitates subsequent processing.

[0073] Step S30: During model training, the frozen CLIP visual encoder is used as the teacher model and the preset cross-modal feature fusion network is used as the student model to train the corresponding small target region mask, and the distillation loss of the dynamically corrected feature map is calculated based on the small target region mask.

[0074] It's important to note that during the model training phase, a knowledge distillation strategy is employed: a frozen CLIP visual encoder is used as the teacher model (CLIP is a pre-trained cross-modal model with strong open-vocabulary understanding capabilities), and a pre-defined cross-modal feature fusion network is used as the student model. Small target region masks are generated during training (used to focus on small target features and address the issue of low detection accuracy for small targets). Based on these masks, the distillation loss of the dynamically corrected feature map is calculated. Specifically, the distillation loss allows the student model to learn the cross-modal semantic knowledge of the teacher model, improving its generalization ability to unseen categories, thus facilitating subsequent processing.

[0075] Step S40: Output multi-scale features corresponding to the feature map based on the distillation loss, and perform bounding box regression and open vocabulary classification on the multi-scale features using a preset detection head to complete the corresponding target detection.

[0076] It should be noted that, finally, based on the distillation loss optimization, the multi-scale features are output, and bounding box regression (determining target location and size) and open vocabulary classification (identifying target categories, including those not seen during training) are completed using a pre-set detection head. This ultimately achieves open vocabulary target detection. Specifically, compared to traditional detection methods that can only identify predefined categories, this method, through cross-modal fusion and knowledge distillation, can handle target categories described in any natural language, facilitating subsequent processing.

[0077] Second Embodiment

[0078] Furthermore, the step of injecting each of the cue anchor vectors into its corresponding feature map via residual connections to dynamically correct the multi-scale semantic bias of the feature map includes:

[0079] Once the feature map is determined, a cue anchor vector corresponding to the feature map is defined;

[0080] The original vector size of the prompt anchor vector is expanded to the same target vector size as the feature map size through a broadcast operation, and the corrected feature map is obtained by adding the residuals.

[0081] It should be noted that, firstly, when a feature map is determined (such as an 8×8 resolution feature map output by a certain layer in a cross-modal network), a corresponding cue anchor vector is defined: the vector dimension is consistent with the number of channels of the feature map (e.g., if the feature map is 8×8×256, then the cue anchor vector is 1×1×256). The initial value of the vector is generated through random initialization. Subsequently, during training, the key semantic features at this scale are learned (e.g., the 8×8 scale may correspond to the global semantics of "vehicle", and the vector will gradually encode key information such as "front of the car" and "body of the car").

[0082] Subsequently, the original vector size (1×1×256) of the cue anchor vector is expanded to a target vector size (8×8×256) that is the same as the feature map size through a broadcast operation. Broadcasting allows a single vector to be copied to every spatial location in the feature map, ensuring that each location is corrected for semantic bias (e.g., each pixel in the 8×8 feature map incorporates the global semantics of "vehicle"). Finally, the expanded vector is combined with the original feature map through residual addition (corrected feature map = original feature map + expanded cue vector). The advantage of residual connection is that it preserves the detailed information of the original features (such as edges and textures) while injecting semantic bias through the cue vector, making the semantics of the feature map more consistent across different scales (e.g., a high-scale feature map retains the details of "wheels" while also associating with the global semantics of "vehicle"), providing a more reliable feature foundation for subsequent classification and regression, thus facilitating subsequent processing.

[0083] Furthermore, the step of training the corresponding small target region mask includes:

[0084] During model training, corresponding annotation information is added to the feature map;

[0085] Based on the annotation information, an initial region with an area smaller than a preset area threshold and an anchor frame intersection-union ratio greater than a preset intersection-union ratio threshold is selected from within the feature map;

[0086] The initial region is set as the small target region, and a corresponding binary mask is generated to set the binary mask as the small target region mask.

[0087] It should be noted that, firstly, during the model training process, corresponding annotation information is added to the feature map: the annotation information includes the bounding box coordinates of the target, the category label (such as "small potted plant", "remote control" and other small targets), and the area of ​​each target (calculated based on the bounding box). This information is used for the subsequent recognition of small target regions.

[0088] Secondly, the initial regions are selected based on the annotation information: a preset area threshold (e.g., an area ≤ 32 pixels² on the feature map, corresponding to small targets in the original image) and a preset intersection-union (IoU) threshold (e.g., the IoU between the anchor box and the labeled bounding box is ≥ 0.5, ensuring that the anchor box accurately covers the target). Regions in the feature map that simultaneously satisfy the criteria of "area less than the area threshold" and "anchor box IoU greater than the IoU threshold" are selected as the initial regions. Specifically, these regions correspond to small targets in the image. Since small targets have few features and are easily overlooked, this selection step can accurately locate the regions that require special attention.

[0089] Finally, the initial region is set as the small target region, and a corresponding binary mask is generated: the size of the binary mask is consistent with the feature map, the mask value for the small target region is 1 (indicating that it needs to be focused on), and the mask value for other regions is 0 (indicating that it can be de-emphasized). This mask plays a weighting role in the subsequent distillation loss calculation, making the loss function focus more on learning the features of small targets, solving the problem in traditional detection where "large target features dominate training, and small targets have low accuracy", and improving the recognition ability of small targets in open vocabulary detection. This facilitates subsequent processing.

[0090] Furthermore, the step of calculating the distillation loss of the dynamically corrected feature map based on the small target region mask includes:

[0091] The first feature output by the student model is mapped to the same semantic space as the second feature output by the teacher model using a linear projection function.

[0092] Based on the semantic space, the distillation loss is calculated using a preset algorithm according to the first feature and the second feature.

[0093] It should be noted that, firstly, the first feature (dynamically corrected multi-scale feature) output by the student model (pre-set cross-modal feature fusion network) is mapped to the same semantic space as the second feature output by the teacher model (CLIP visual encoder) through a linear projection function. Since the network structures of the student model and the teacher model are different, the dimensions and semantic distribution of the output features may differ. The linear projection function (such as a fully connected layer) can convert the dimension of the first feature to be consistent with that of the second feature (e.g., both are converted to 768 dimensions) and align the semantic distribution of the two (e.g., the feature of "cat" is in a similar semantic space position in the two models), providing a unified benchmark for loss calculation.

[0094] Based on the aligned semantic space, a pre-defined algorithm calculates distillation loss using the first and second features. The core of distillation loss is to reduce the feature differences between the student and teacher models, enabling the student model to learn the teacher model's cross-modal semantic knowledge (such as CLIP's understanding of open-ended terms like "unicorn" and "alien creature"). The pre-defined algorithm must simultaneously consider global feature differences and local differences in small target regions (combined with the aforementioned small target region mask) to ensure that the student model not only learns the overall semantics but also accurately captures the detailed features of small targets, ultimately improving its generalization ability to open-ended terms. This facilitates subsequent processing.

[0095] Furthermore, the expression for the preset algorithm is:

[0096]

[0097] Among them, F i (s) For student model features, F i (t) For teacher model features, M is the numerically stable term. i This represents the mask for the small target region.

[0098] Furthermore, the step of performing bounding box regression and open vocabulary classification on the multi-scale features using a preset detection head to complete the corresponding target detection includes:

[0099] The category names or natural language prompts of the multi-scale features are converted into corresponding text embeddings;

[0100] Visual features corresponding to the feature map are extracted from the multi-scale features, and the feature map is classified based on the text embedding and the visual features.

[0101] It's important to note that, firstly, the category names corresponding to multi-scale features (such as "bicycle" and "traffic light") or natural language cues (such as "a two-wheeled vehicle") are converted into text embeddings. Text embeddings are achieved by using a pre-trained text encoder (such as CLIP's text encoder) to convert textual information into high-dimensional vectors (e.g., 768-dimensional). These vectors encode the semantic information of the text (e.g., the embedding of "bicycle" includes semantics such as "wheels," "riding," and "vehicle"). This step is crucial for open-vocabulary detection, enabling the model to handle categories of arbitrary text descriptions and overcoming the limitations of traditional detection methods that rely on predefined categories.

[0102] Secondly, visual features corresponding to the feature maps are extracted from the multi-scale features: visual features are those optimized through dynamic correction and distillation loss, containing image semantics at different scales (such as "overall shape" at a low scale and "local details" at a high scale). Finally, feature map classification is performed based on text embeddings and visual features: by calculating the semantic similarity (such as cosine similarity) between text embeddings and visual features, the best-matching text category is found (e.g., the visual features have the highest similarity to the text embedding of "bicycle"), achieving open-vocabulary classification; simultaneously, the detection head determines the coordinates and size of the target through bounding box regression (based on the positional information of visual features), ultimately completing target detection. This facilitates subsequent processing.

[0103] Furthermore, the step of classifying the feature map based on the text embedding and the visual features includes:

[0104] The cosine similarity between the text embedding and the visual features is calculated, and the feature map is classified based on the magnitude of the cosine similarity.

[0105] It's important to note that, firstly, the cosine similarity between the text embedding and the visual features is calculated: cosine similarity measures the directional consistency between two vectors (range [-1, 1]), and the closer the value is to 1, the better the semantic match between the text and visual features. For example, the cosine similarity between the text embedding of "cat" and the visual features of a cat in an image will be significantly higher than the similarity with the text embedding of "dog".

[0106] Feature map classification is performed based on cosine similarity: For each visual feature (corresponding to a target candidate region in the feature map), its cosine similarity with all text embeddings is ranked, and the text category with the highest similarity is taken as the predicted category for that region. If the highest similarity value is lower than a preset threshold (e.g., 0.5), it is determined as an "unknown category". This cosine similarity-based classification method utilizes the property of "semantic similarity leads to similar vectors" in cross-modal semantic space, enabling the model to recognize categories not seen during training (e.g., having learned "cat" and "dog" during training, it can identify "squirrel" through the text embedding "furry, tree-climbing animal"), truly achieving the generalization ability of open-vocabulary target detection. This facilitates subsequent processing.

[0107] Please refer to Figure 2. The third embodiment of the present invention provides:

[0108] An open vocabulary target detection system, wherein the system comprises:

[0109] An integration module is used to integrate a learnable cue anchor module into the multi-scale feature layers of a preset cross-modal feature fusion network. The learnable cue anchor module is used to define a set of learnable cue anchor vectors for each feature map layer.

[0110] The correction module is used to inject each of the cue anchor vectors into its corresponding feature map through residual connections, so as to dynamically correct the multi-scale semantic bias of the feature map.

[0111] The training module is used to train the corresponding small target region mask by using the frozen CLIP visual encoder as the teacher model and the preset cross-modal feature fusion network as the student model during the model training process, and to calculate the distillation loss of the dynamically corrected feature map based on the small target region mask.

[0112] The classification module is used to output multi-scale features corresponding to the feature map based on the distillation loss, and to perform bounding box regression and open vocabulary classification on the multi-scale features through a preset detection head to complete the corresponding target detection.

[0113] Furthermore, the correction module is specifically used for:

[0114] Once the feature map is determined, a cue anchor vector corresponding to the feature map is defined;

[0115] The original vector size of the prompt anchor vector is expanded to the same target vector size as the feature map size through a broadcast operation, and the corrected feature map is obtained by adding the residuals.

[0116] Furthermore, the training module is specifically used for:

[0117] During model training, corresponding annotation information is added to the feature map;

[0118] Based on the annotation information, an initial region with an area smaller than a preset area threshold and an anchor frame intersection-union ratio greater than a preset intersection-union ratio threshold is selected from within the feature map;

[0119] The initial region is set as the small target region, and a corresponding binary mask is generated to set the binary mask as the small target region mask.

[0120] Furthermore, the training module is specifically used for:

[0121] The first feature output by the student model is mapped to the same semantic space as the second feature output by the teacher model using a linear projection function.

[0122] Based on the semantic space, the distillation loss is calculated using a preset algorithm according to the first feature and the second feature.

[0123] Furthermore, the expression for the preset algorithm is:

[0124]

[0125] Among them, F i (s) For student model features, F i (t) For teacher model features, M is the numerically stable term. i This represents the mask for the small target region.

[0126] Furthermore, the classification module is specifically used for:

[0127] The category names or natural language prompts of the multi-scale features are converted into corresponding text embeddings;

[0128] Visual features corresponding to the feature map are extracted from the multi-scale features, and the feature map is classified based on the text embedding and the visual features.

[0129] Furthermore, the classification module is specifically used for:

[0130] The cosine similarity between the text embedding and the visual features is calculated, and the feature map is classified based on the magnitude of the cosine similarity.

[0131] The fourth embodiment of the present invention provides a computer, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the open vocabulary target detection method as described above.

[0132] The fifth embodiment of the present invention provides a readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the open vocabulary target detection method as described above.

[0133] In summary, the open vocabulary target detection method and system provided by the above embodiments of the present invention can accurately and effectively detect features within small target regions, thereby effectively completing the classification of feature maps and improving detection efficiency.

[0134] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0135] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0136] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0137] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0138] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0139] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. An open vocabulary target detection method, characterized in that, The method includes: integrating a learnable cue anchor module into the multi-scale feature layers of a pre-defined cross-modal feature fusion network; the learnable cue anchor module is used to define a set of learnable cue anchor vectors for each feature map layer, and the cue anchor vectors are used to capture key semantic information at each scale; injecting each cue anchor vector into its corresponding feature map through residual connections to introduce semantic bias of the cue anchors, so as to dynamically correct the multi-scale semantic bias of the feature maps; during model training, using a frozen CLIP visual encoder as the teacher model and the pre-defined cross-modal feature fusion network as the learner model. The model is trained to generate a corresponding small target region mask, and the distillation loss of the dynamically corrected feature map is calculated based on the small target region mask. Multi-scale features corresponding to the feature map are output based on the distillation loss, and bounding box regression and open vocabulary classification are performed on the multi-scale features using a preset detection head to complete the corresponding target detection. The step of injecting each cue anchor vector into its corresponding feature map through residual connections to dynamically correct the multi-scale semantic bias of the feature map includes: when the feature map is determined, defining a cue anchor vector corresponding to the feature map; wherein... The vector dimension of the cue anchor vector is consistent with the number of channels of the feature map. The initial value of the cue anchor vector is generated through random initialization and is used to learn key semantic features at various scales during training. The original vector size of the cue anchor vector is expanded to the same target vector size as the feature map size through a broadcast operation, and the corrected feature map is obtained by adding the residuals. The step of training the corresponding small target region mask includes: adding corresponding annotation information to the feature map during model training; selecting initial regions with an area smaller than a preset area threshold and an intersection-union ratio (IU) of anchor boxes greater than a preset IU threshold within the feature map based on the annotation information; setting the initial regions as the small target regions and generating corresponding binary masks to set the binary masks as the small target region masks. The step of calculating the distillation loss of the dynamically corrected feature map based on the small target region mask includes: mapping the first feature output by the student model to the same semantic space as the second feature output by the teacher model through a linear projection function; calculating the distillation loss based on the semantic space using a preset algorithm according to the first feature and the second feature; the expression of the preset algorithm is: Among them, F i (s) For student model features, F i (t) For teacher model features, M is the numerically stable term. i This represents the mask for the small target region.

2. The open vocabulary target detection method according to claim 1, characterized in that, The step of performing bounding box regression and open vocabulary classification on the multi-scale features using a preset detection head to complete the corresponding target detection includes: converting the category name or natural language prompt of the multi-scale features into the corresponding text embedding; extracting the visual features corresponding to the feature map from the multi-scale features; and completing the classification of the feature map based on the text embedding and the visual features.

3. The open vocabulary target detection method according to claim 2, characterized in that, The step of classifying the feature map based on the text embedding and the visual features includes: calculating the cosine similarity between the text embedding and the visual features, and classifying the feature map based on the magnitude of the cosine similarity.

4. An open vocabulary target detection system, characterized in that, To implement the open vocabulary target detection method as described in any one of claims 1 to 3, the system comprises: an integration module, used to integrate a learnable cue anchor module into a multi-scale feature layer of a preset cross-modal feature fusion network, wherein the learnable cue anchor module is used to define a set of learnable cue anchor vectors for each feature map layer; a correction module, used to inject each of the cue anchor vectors into its corresponding feature map through residual connections to dynamically correct the multi-scale semantic bias of the feature map; a training module, used during model training to train a corresponding small target region mask using a frozen CLIP visual encoder as a teacher model and the preset cross-modal feature fusion network as a student model, and to calculate the distillation loss of the dynamically corrected feature map based on the small target region mask; and a classification module, used to output multi-scale features corresponding to the feature map based on the distillation loss, and to perform bounding box regression and open vocabulary classification on the multi-scale features using a preset detection head to complete the corresponding target detection.

5. A computer, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the open vocabulary target detection method as described in any one of claims 1 to 3.

6. A readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the open vocabulary target detection method as described in any one of claims 1 to 3.

Images