An open vocabulary detection method and device based on background mining and a medium

Abstract

This invention relates to an open vocabulary detection method, device, and medium based on background mining. The method involves inputting a background candidate set into a CLIP text encoder to generate an initial background prototype by encoding preset learnable background tags; extracting pixel-level features of the entire image using an encoder with an arbitrary segmentation model; fusing the initial background prototype and pixel-level features through a multi-head cross-attention module to output an enhanced background prototype; inputting a foreground candidate set into a CLIP image encoder to extract foreground features, and generating a foreground-enhanced background prototype through a multi-head cross-attention module; calculating the cosine similarity between each region in the background candidate set and the foreground-enhanced background prototype; assigning extended category labels to the background candidate set; and outputting the open vocabulary detection results. Compared with existing technologies, this invention has advantages such as good generalization performance, strong cross-modal understanding ability, and strong robustness.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and open vocabulary detection, and in particular to an open vocabulary detection method, device and medium based on background mining. Background Technology

[0002] Object detection techniques aim to locate objects in images and assign semantic labels. While modern detectors have made significant progress in scenarios with closed-category sets, their reliance on fixed predefined categories severely limits their ability to generalize to new objects. This limitation has driven the development of Open-vocabulary Object Detection (OVD), a technique that aims to detect both seen and unseen categories during training without requiring explicit labeling of new categories. Current mainstream OVD methods generally employ pre-trained visual language models (PVLMs) to achieve cross-modal knowledge transfer; however, significant drawbacks remain: existing schemes typically simplify complex background regions into a single "background" class learnable embedding. This abstraction ignores the rich and diverse semantics within the background region, which are crucial clues for uncovering new objects beyond closed categories. Furthermore, most methods adopt a fragmented approach when dealing with foreground (containing known objects) and background regions, failing to leverage foreground semantics to enhance background understanding. This results in the model's inability to effectively capture potential information in the background that has semantic connections with proposed known objects. The aforementioned shortcomings collectively reflect the core contradiction of the current OVD paradigm. On the one hand, because the background is compressed into a single-category representation, the semantic diversity of the background region is not fully explored, weakening the model's ability to discover new categories. On the other hand, the lack of a mechanism for foreground information to guide background understanding results in insufficient modeling of cross-modal space-semantic relationships. Therefore, how to fully utilize the semantics of the background region in image detection to more accurately identify new objects in open-vocabulary scenarios is a technical problem that needs to be solved. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the existing technology by providing an open vocabulary detection method, device and medium based on background mining. By establishing a foreground-background semantic collaboration mechanism, the background region and the foreground region can participate more accurately in the target detection process, opening up the semantic interaction channel between the foreground and background regions, and improving the model's ability to detect potential semantic clues related to known objects in the background.

[0004] The objective of this invention can be achieved through the following technical solutions:

[0005] According to one aspect of the present invention, an open vocabulary detection method based on background mining is provided, the specific steps of which include:

[0006] S1. Input the image to be detected into the backbone network and the region proposal network to extract a candidate region set; the candidate region set includes a foreground candidate set containing basic class instances and a background candidate set not containing basic class instances;

[0007] S2. Input the background candidate set into the CLIP text encoder, generate an initial background prototype by encoding a preset learnable background marker; extract pixel-level features of the entire image using an encoder with an arbitrary segmentation model; fuse the initial background prototype and pixel-level features through a multi-head cross-attention module to output an enhanced background prototype.

[0008] S3. Input the foreground candidate set into the CLIP image encoder to extract foreground features, and generate a foreground-enhanced background prototype through a multi-head cross-attention module;

[0009] S4. Calculate the cosine similarity between each region in the background candidate set and the foreground enhanced background prototype; assign extended category labels to the background candidate set based on the similarity, and combine the basic category labels of the foreground candidate set to output the open vocabulary detection results.

[0010] Furthermore, the multi-head cross-attention module in S2 performs a first fusion, using the initial background prototype as the query and pixel-level features as the key and value, expressed as:

[0011] P′ bg =MHCA(P bg ,F SAM ,F SAM );

[0012] Among them, P′ bg To enhance the background prototype, P bg As the background prototype, F SAM These are pixel-level features.

[0013] Furthermore, the expression for the multi-head attention mechanism in the first fusion is:

[0014] MHCA(P bg ,F SAM ,F SAM = Concat(head1,…,head) h W i 1,

[0015]

[0016] Where d h = d / h is the dimension of each head, where d is the feature dimension and h is the number of attention heads. W is the projection matrix of the i-th head. o 1 represents the output projection matrix of the first fusion, headi This is the output of the first fusion of each attention head.

[0017] Furthermore, the multi-head cross-attention module in S3 performs a second fusion, using the enhanced background prototype as the query and the foreground features as the key and value, expressed as:

[0018]

[0019] Among them, P″ bg To enhance the foreground prototype, P′ bg To enhance the background prototype, Foreground features.

[0020] Furthermore, the expression for the multi-head attention mechanism in the second fusion is:

[0021]

[0022] Where d h = d / h is the dimension of each head, where d is the feature dimension and h is the number of attention heads. W is the projection matrix of the i-th head. o 2 represents the output projection matrix of the second fusion, head i ′ represents the output of the second fusion for each attention head.

[0023] Furthermore, in step S4, the classification probability of each region in the background candidate set on the extended category set is calculated based on the cosine similarity, and the pseudo-label corresponding to the extended category with the highest classification probability is selected.

[0024] The expression for the classification probability is:

[0025]

[0026] Where x is the region in the candidate background set to be classified, c is the candidate category label, and P″ bg To enhance the background prototype for the foreground, e c c represents the prototype vector of each region in the background candidate set. u τ is a hyperparameter used to expand the category set.

[0027] Furthermore, the extended category set includes a background prototype category set and a general background category set. When the cosine similarity is higher than the threshold, the semantic pattern corresponding to the prototype vector is a label in the background prototype category set; otherwise, it is a label in the general background category set.

[0028] Furthermore, the joint loss function for system training of the method includes foreground classification loss, foreground-guided prototype contrast loss, dynamic pseudo-label confidence adaptation loss, and relaxed soft background loss;

[0029] The foreground classification loss L cls The expression is:

[0030]

[0031] in, Let T(x) be the foreground candidate set, T(x) be the true base class label of region x, and p(T(x)|x) be the class probability distribution of the true base class label T(x) in region x.

[0032] The foreground-guided prototype contrast loss includes inter-prototype contrast loss and intra-category compactness loss, wherein the inter-prototype contrast loss L inter The expression is:

[0033]

[0034] Among them, P″ bg To enhance the foreground prototype, m is the boundary constraint value, t c The CLIP text embedding vector based on category c, Basic category set;

[0035] The compactness loss within the category L intra The expression is:

[0036]

[0037]

[0038] Among them, Ω(P″) bg f represents the set of background suggestions related to the foreground enhancement background prototype. j CLIP visual features of background region j The centroid of the suggested set for the background;

[0039] The dynamic pseudo-label confidence adaptation loss The expression is:

[0040]

[0041] Where, λ bg To balance the coefficients, the relative penalty for proposals in low-confidence backgrounds is controlled. For high-confidence background regions aligned with pseudo-labels, It is the remaining region, y o (x) is a pseudo-label, c bg Background category label;

[0042] The relaxed soft background loss L rlx The expression is

[0043]

[0044] in, For regions where the maximum similarity is below a preset threshold γ, The number of background prototypes.

[0045] According to a second aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described thereon.

[0046] According to a third aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described thereon.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] (1) Achieve fine-grained modeling of background semantics and improve the ability to discover new categories: Generate dynamic initial background prototypes by learning background tags, and perform semantic fusion by combining the full-image pixel-level features extracted by arbitrary segmentation models to form a polysemous enhanced background prototype; further introduce an extended category set that includes background prototype classes and general background classes, break through the limitation of traditional methods that simplify the background to a single embedding, realize the parsing modeling of diverse semantics of background regions in open vocabulary detection, provide key clues for discovering new objects outside the closed category, and effectively solve the defect of insufficient utilization of background semantics.

[0049] (2) Establish a foreground-background semantic collaboration mechanism to enhance cross-modal understanding: Utilize a multi-head cross attention module to enhance the background prototype as the query and the foreground region features as the key value to realize the dynamic guidance and optimization of the background prototype by the foreground semantic information, open up the semantic interaction channel between the foreground and background regions, improve the model's ability to capture potential semantic clues (such as local features of objects and occlusion structures) associated with known objects in the background, and overcome the semantic fragmentation problem caused by isolated processing of foreground and background in the existing technology.

[0050] (3) Adaptive pseudo-label allocation and joint optimization to improve the robustness of open vocabulary: The semantic similarity of the background region is calculated based on the foreground enhancement background prototype, and the adaptive allocation of extended category labels is achieved through the probability distribution adjusted by the temperature coefficient; combined with the joint loss function containing multi-level supervision signals (foreground classification loss, prototype contrast loss, dynamic confidence loss, soft background regularization loss), a differentiated optimization strategy is implemented for background regions with different confidence levels to enhance the robustness of the model in identifying unknown categories and the noise resistance of complex scenes, and solve the problem of ambiguous background region labeling in open vocabulary detection. Attached Figure Description

[0051] Figure 1 This is a flowchart of an open vocabulary detection method based on background mining. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0053] Traditional object detection methods typically pay little attention to background regions. However, the enhanced context aggregator-foreground-enhanced prototype-guided pseudo-label assigner framework in this embodiment guides background prototype mining through foreground, enabling background regions to participate more accurately in the object detection process and improving the model's understanding and recognition of background semantics. This method is particularly suitable for open-vocabulary detection scenarios, extracting more semantic information from diverse backgrounds and improving the recognition and classification of background semantics.

[0054] Furthermore, by combining the Segment Anything model with CLIP's encoder, the enhanced context aggregator module not only generates background prototypes but also effectively fuses textual and pixel-level features, enabling fine-grained modeling of diverse background semantics. Unlike traditional single-background embedding methods, the enhanced context aggregator provides richer and more refined semantic information, allowing the model to understand complex background scenes. In image processing tasks, especially when image backgrounds are complex or varied, this innovative design effectively improves the accuracy of background region recognition and enhances the model's ability to identify objects under different backgrounds.

[0055] The foreground-enhanced prototype-guided pseudo-label assigner, by introducing semantic cues enhanced by the foreground and incorporating a cross-attention mechanism, enables effective interaction between foreground and background regions. Traditional object detection methods often treat foreground and background regions in isolation, neglecting their semantic connections. FEPLA, however, refines the background prototype, enhancing not only the background region's ability to assign pseudo-labels but also facilitating the discovery of novel categories. This innovation improves the model's adaptability to open-vocabulary object detection tasks, particularly in discovering unknown categories, by leveraging foreground features to infer and optimize background labels, thereby enhancing the detection capability for new categories.

[0056] like Figure 1 As shown, this embodiment provides an open vocabulary detection method based on background mining, the specific steps of which include:

[0057] S1. Input the image to be detected into the backbone network and the region proposal network to extract a candidate region set; the candidate region set includes a foreground candidate set containing basic class instances and a background candidate set not containing basic class instances.

[0058] S2. Input the background candidate set into the CLIP text encoder to generate an initial background prototype by encoding the preset learnable background markers; extract pixel-level features of the entire image using an encoder with an arbitrary segmentation model; fuse the initial background prototype and pixel-level features through a multi-head cross-attention module to output an enhanced background prototype.

[0059] S3. Input the foreground candidate set into the CLIP image encoder to extract foreground features, and generate a foreground-enhanced background prototype through a multi-head cross-attention module;

[0060] S4. Calculate the cosine similarity between each region in the background candidate set and the foreground enhanced background prototype; assign extended category labels to the background candidate set based on the similarity, and combine the basic category labels of the foreground candidate set to output the open vocabulary detection results.

[0061] S2 uses the CLIP text encoder Enc T (·) For learnable background labels V bg Encode to initialize the background prototype P bg The expression is:

[0062] P bg =Enc T (V bg ).

[0063] The SAM-enhanced Context Aggregator (SAM-CA) extracts features from the complete image using an encoder based on the Segment Anything Model (SAM). These pixel-level features are compared with the background prototype P obtained from the CLIP text encoder. bg Fusion enables the model to construct a richer semantic space for the background region. Through the multi-head cross-attention module MHCA(·), which systematically aggregates contextual cues, the information of the background prototype is further enhanced, resulting in an enhanced background prototype P′. bg .

[0064] Given an input image I, extract pixel-level features F using the encoder of an arbitrary segmentation model SAM. SAM Let Enc SAM (·) represents the SAM encoder; the extracted feature expression is:

[0065] F SAM =Enc SAM (I).

[0066] Pixel-level features F SAM Capturing dense, fine-grained semantic representations is crucial for detailed contextual understanding.

[0067] To optimize the background prototype, a multi-head cross-attention (MHCA) mechanism was employed, denoted as MHCA(·). The first fusion was performed using a multi-head cross-attention module, taking the initial background prototype as the query and pixel-level features as keys and values. Fine-grained semantic cues were injected into the background prototype, generating a richer, enhanced representation. This enhanced background prototype captures a wider range of latent semantics from the background region, providing a stronger foundation for subsequent optimization and new category discovery. The expression is:

[0068] P′ bg =MHCA(P bg ,F SAM ,F SAM );

[0069] Among them, P′ bg To enhance the background prototype, P bg As the background prototype, F SAM These are pixel-level features.

[0070] The expression for the multi-head attention mechanism in the first fusion is:

[0071] MHCA(P bg ,f SAM ,f SAM = Concat(head1,…,head) h W o 1,

[0072]

[0073] Where d h = d / h is the dimension of each head, where d is the feature dimension and h is the number of attention heads. W is the projection matrix of the i-th head. o 1 represents the output projection matrix of the first fusion, head i This is the output of the first fusion of each attention head.

[0074] By incorporating semantic cues from the foreground region, the enhanced background prototype generated by SAM-CA is expanded.

[0075] set up

[0076]

[0077] This represents the set of regions based on RoI, where N is the number of foreground regions. Each region r jAll through the frozen CLIP image encoder Enc V (·) Processing to produce foreground embedding:

[0078] f j =Enc V (r j ).

[0079] The encoded features are used as the foreground feature set.

[0080]

[0081] The multi-head cross-attention module in S3 performs a second fusion, using the enhanced background prototype as the query and the foreground features as the key and value, expressed as:

[0082]

[0083] Among them, P″ bg To enhance the foreground prototype, P′ bg To enhance the background prototype, Foreground features.

[0084] Furthermore, the expression for the multi-head attention mechanism in the second fusion is:

[0085]

[0086] Where d h = d / h is the dimension of each head, where d is the feature dimension and h is the number of attention heads. W is the projection matrix of the i-th head. o 2 represents the output projection matrix of the second fusion, head i ′ represents the output of the second fusion for each attention head.

[0087] In S4, the classification probability of each region in the background candidate set on the extended category set is calculated based on the cosine similarity, and the pseudo-label corresponding to the extended category with the highest classification probability is selected.

[0088] The expression for the classification probability is:

[0089]

[0090] Where x is the region in the candidate background set to be classified, c is the candidate category label, and P″ bg To enhance the background prototype for the foreground, e c c represents the prototype vector of each region in the background candidate set. u τ is a hyperparameter used to expand the category set.

[0091] The extended category set includes the background prototype category set and the general background category set. When the cosine similarity is higher than the threshold, the semantic pattern corresponding to the prototype vector is the label in the background prototype category set; otherwise, it is the label in the general background category set.

[0092] The joint loss function for system training in this embodiment includes foreground classification loss, foreground-guided prototype contrast loss, dynamic pseudo-label confidence adaptation loss, and relaxed soft background loss.

[0093] Foreground classification loss L cls The expression is:

[0094]

[0095] in, Let T(x) be the foreground candidate set, T(x) be the true base class label of region x, and p(T(x)|x) be the class probability distribution of the true base class label T(x) in region x.

[0096] The foreground-guided prototype contrastive loss includes inter-prototype contrastive loss and intra-category compactness loss, encouraging inter-category separability between discovered background prototypes and the base category, while also promoting intra-category compactness among proposals associated with the same potential category. Foreground-guided prototype L FPC The expression is:

[0097] L FPC =λ inter L inter +λ intra L intra ,

[0098] Where, λ inter and λ intra These are weighting coefficients, which balance the contributions of inter-category separability and intra-category compactness, respectively.

[0099] Inter-prototype contrast loss L inter The expression is:

[0100]

[0101] Among them, P″ bg To enhance the foreground prototype, m is the boundary constraint value, t c The CLIP text embedding vector based on category c, Basic category set;

[0102] Intra-category compactness loss L intra The expression is:

[0103]

[0104] Among them, Ω(P″) bg f represents the set of background suggestions related to the foreground enhancement background prototype. j CLIP visual features of background region j The centroid of the suggested set for the background;

[0105] The region is divided into two disjoint sets: a high-confidence background region aligned with the pseudo-labels. and the remaining area Dynamic pseudo-label confidence adaptation loss The expression is:

[0106]

[0107] Where, λ bg To balance the coefficients, the relative penalty for proposals in low-confidence backgrounds is controlled. For high-confidence background regions aligned with pseudo-labels, It is the remaining region, y o (x) is a pseudo-label, c bg Background category label;

[0108] To further address uncertain cases with extremely low semantic confidence, a relaxed soft background loss L is introduced. rlx For background regions whose maximum similarity to any background prototype is below a small threshold γ, the model is encouraged to gently align them to all background-related prototypes through uniform regularization. Relaxed soft background loss L rlx The expression is:

[0109]

[0110] in, For regions where the maximum similarity is below a preset threshold γ, The number of background prototypes.

[0111] Total loss function L fianl The expression is:

[0112] L fianl =L cls +L FPC +L DPCA +L rlx .

[0113] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0114] The electronic device of this invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) or loaded from a storage unit into random access memory (RAM). The RAM may also store various programs and data required for device operation. The CPU, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0115] Multiple components in the device are connected to an I / O interface, including: input units such as a keyboard, mouse, etc.; output units such as various types of displays, speakers, etc.; storage units such as disks, optical disks, etc.; and communication units such as network interface cards, modems, wireless transceivers, etc. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks. The processing unit performs the various methods and processes described above, such as the method of the present invention. For example, in some embodiments, the method of the present invention may be implemented as a computer software program tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via ROM and / or the communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of the method of the present invention described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute the method of the present invention by any other suitable means (e.g., by means of firmware).

[0116] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0117] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0118] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0119] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An open vocabulary detection method based on background mining, characterized in that, The specific steps include: S1. Input the image to be detected into the backbone network and the region proposal network to extract a candidate region set; the candidate region set includes a foreground candidate set containing basic class instances and a background candidate set not containing basic class instances; S2. Input the background candidate set into the CLIP text encoder, generate an initial background prototype by encoding a preset learnable background marker; extract pixel-level features of the entire image using an encoder with an arbitrary segmentation model; fuse the initial background prototype and pixel-level features through a multi-head cross-attention module to output an enhanced background prototype. S3. Input the foreground candidate set into the CLIP image encoder to extract foreground features, and generate a foreground-enhanced background prototype through a multi-head cross-attention module; S4. Calculate the cosine similarity between each region in the background candidate set and the foreground enhanced background prototype; assign extended category labels to the background candidate set based on the similarity, and combine the basic category labels of the foreground candidate set to output the open vocabulary detection results; The joint loss function for system training of the method includes foreground classification loss, foreground-guided prototype contrast loss, dynamic pseudo-label confidence adaptation loss, and relaxed soft background loss. The foreground classification loss The expression is: ， in, For the foreground candidate set, For the region x The actual basic category labels, For real base category labels In the region x The category probability distribution; The foreground-guided prototype contrast loss includes inter-prototype contrast loss and intra-category compactness loss. The inter-prototype contrast loss... The expression is: in, Enhance the background prototype to enhance the foreground. These are boundary constraint values. The CLIP text embedding vector based on category c, Basic category set; Compactness loss within the category The expression is: ， ， in, A set of background suggestions related to the foreground enhancement background prototype. CLIP visual features of background region j The centroid of the suggested set for the background; The dynamic pseudo-label confidence adaptation loss The expression is: ， in, To balance the coefficients, the relative penalty for proposals in low-confidence backgrounds is controlled. For high-confidence background regions aligned with pseudo-labels, It is the remaining area. It is a pseudo-label. Background category label; The relaxed soft background loss The expression is ， in, For maximum similarity below a preset threshold The area , The number of background prototypes.

2. The open vocabulary detection method based on background mining according to claim 1, characterized in that, The multi-head cross-attention module in S2 performs the first fusion, using the initial background prototype as the query and pixel-level features as the key and value, expressed as: ； in, To enhance the background prototype, As the background prototype, These are pixel-level features.

3. The open vocabulary detection method based on background mining according to claim 2, characterized in that, The expression for the multi-head attention mechanism in the first fusion is: ， ， in, For the dimensions of each head in a multi-head attention mechanism, d For feature dimension, h For the number of attention heads, It is the first The projection matrix of the head, The output projection matrix of the first fusion is This is the output of the first fusion of each attention head.

4. The open vocabulary detection method based on background mining according to claim 1, characterized in that, The multi-head cross-attention module in S3 performs a second fusion, using the enhanced background prototype as the query and the foreground features as the key and value, expressed as: ； in, Enhance the background prototype to enhance the foreground. To enhance the background prototype, Foreground features.

5. The open vocabulary detection method based on background mining according to claim 4, characterized in that, The expression for the multi-head attention mechanism in the second fusion is: ， ， in For each head dimension, d For feature dimension, h For the number of attention heads, It is the first The projection matrix of the head, The output projection matrix of the second fusion. This is the output of each attention head in the second fusion.

6. The open vocabulary detection method based on background mining according to claim 1, characterized in that, In step S4, the classification probability of each region in the background candidate set on the extended category set is calculated based on the cosine similarity, and the pseudo-label corresponding to the extended category with the highest classification probability is selected. The expression for the classification probability is: ， in, The region is the candidate set of background elements to be classified. For candidate category labels, Enhance the background prototype to enhance the foreground. These are the prototype vectors of each region in the background candidate set. To expand the category set, This is a hyperparameter.

7. The open vocabulary detection method based on background mining according to claim 6, characterized in that, The extended category set includes a background prototype category set and a general background category set. When the cosine similarity is higher than the threshold, the semantic pattern corresponding to the prototype vector is a label in the background prototype category set; otherwise, it is a label in the general background category set.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 7.

Images