Zero-shot detection method based on visual-linguistic pre-training model and category prototype

Abstract

This application specifically relates to a zero-shot target detection method based on a visual-language pre-trained model and category prototypes. The method includes: after extracting the region of interest (ROI) of the target to be detected, extracting region features from the ROI and inputting them into a pre-set visual pre-trained model to obtain a visual embedding vector; inputting the text information of the base class and novel class into a pre-set text pre-trained model to obtain a text embedding vector; constructing a category based on the classification scores of the base class and novel class; and generating a classification result for the target to be detected based on the alignment results of the visual embedding vector and the text embedding vector, and the similarity results between the ROI and the feature-constructed category. This solves the problems of high dataset annotation costs, high resource consumption, and a lack of optimization methods for the pre-trained model during pre-training, which reduce the accuracy of zero-shot target detection.

Description

Zero-shot detection method based on vision-language pre-trained model and category prototype Technical Field

[0001] This invention relates to the field of object detection technology, and in particular to a zero-shot detection method based on a visual-language pre-trained model and a category prototype. Background Technology

[0002] The series of works on visual-language models for classification tasks mainly revolves around CLIP (Contrastive Language-Image Pre-training), including CoOp, CoCoOp, CLIP-Adapter, and Tip-Adapter. This part of the work uses a large-scale image caption dataset for pre-training to obtain visual-language models with strong generalization capabilities. Building upon this, improvements are mostly made from the perspectives of prompt-tuning and adapter design, thereby enhancing the classification performance of the corresponding methods on specific datasets.

[0003] In related technologies, visual-language model series works for (zero-shot) detection tasks can be broadly divided into CLIP-like, including ViLD, DetProh, VL-PLM, etc., and GLIP-like, including GLIP, GLIPv2, DetCLIP, etc. Among them, CLIP-like uses a large-scale image caption dataset for pre-training, while GLIP-like uses a grounding data dataset of the same size for pre-training. Compared with the image caption dataset, the grounding data dataset does not contain image-level descriptive information, but it contains object bounding box-level annotations.

[0004] Tip-adapter is a classic approach to image classification using a visual-language model, designed from an adapter design perspective. This method uses fragmented-shot data, generates a set of features via the CLIP model's image encoder, and uses these features as prototypes for different categories. Finally, classification is achieved by calculating the similarity between the feature information of the image to be classified and the prototypes of different categories. Meanwhile, methods that utilize category prototypes to improve image classification or object detection have also gained widespread attention, extending beyond the use of visual-language models. For example, methods such as Few-Shot Object Detection by Attending to Per-Sample-Prototype, MA-GCP, and FSODup are used to design prototypes based on single sample data or local object features for fragmented-shot or semi-supervised detection tasks, thereby improving object detection performance.

[0005] However, the series of works on GLIP-like types have the following drawbacks: (1) They are difficult to apply directly on the edge. At the same time, GLIP-like models are too large, so they will consume a lot of resources and take a long time to train; (2) The model pre-training stage uses a large-scale Grounding Dataset, which results in high annotation costs; (3) GLIP-like methods perform worse than CLIP-like methods on Open Vocabulary Detection tasks.

[0006] The series of works targeting CLIP-like types have the following drawbacks: (1) No relevant work has been found that utilizes the idea of ​​Adapter or category Prototype design to improve the detection accuracy of existing models; (2) The relevant work on category Prototype improvement ignores the zero-shot object detection problem and only focuses on Few-shot or supervised image classification or object detection tasks. At the same time, there is a lack of optimization strategies specifically for vision-language pre-trained models, which urgently need to be addressed. Summary of the Invention

[0007] This application provides a zero-shot detection method based on a visual-language pre-trained model and category prototypes to address the problems of high dataset annotation costs, high resource consumption, long processing time, and lack of optimization methods for visual-language pre-trained models during the pre-training process, which reduce the accuracy of zero-shot object detection.

[0008] The first aspect of this application provides a zero-shot detection method based on a vision-language pre-trained model and a category prototype, comprising the following steps:

[0009] Obtain the target to be detected, extract the region of interest of the target to be detected, and extract volume-level region features from the region of interest;

[0010] The object-level region features are input into a preset visual pre-trained model to obtain visual embedding vectors, and the text information of the base class and novel class are input into a preset text pre-trained model to obtain text embedding vectors; and

[0011] Based on the visual embedding vector, feature-constructed categories are obtained according to the classification score regions of the base class and the novel class. The classification result of the target to be detected is generated according to the alignment result of the visual embedding vector and the text embedding vector and the similarity result between the region of interest and the feature-constructed categories.

[0012] According to one embodiment of this application, extracting the region of interest of the target to be detected and extracting volume-level region features from the region of interest includes:

[0013] Based on the target to be detected, visual weight information in the target to be detected is obtained to obtain the visual feature value of the target to be detected;

[0014] The Region Proposal Network (RPN) extracts the region of interest (ROI) of the target object based on the visual feature values, and multiple cascaded RoIAlign modules extract object-level region features that meet preset irrelevance conditions.

[0015] According to one embodiment of this application, before inputting the text information of the base class and the novel class into a preset text pre-trained model to obtain the text embedding vector, the method further includes:

[0016] Based on the target to be detected, the text weight information in the target to be detected is obtained, and the text feature value of the target to be detected is obtained.

[0017] The text information of the base class and the novel class is extracted based on the text feature values.

[0018] According to one embodiment of this application, a category is constructed by obtaining features based on the classification score region of the base class and the classification score region of the novel class, including:

[0019] Based on the visual embedding vector, the classification score region of the base class is selected through a first preset learning method to obtain the feature construction category of the base class;

[0020] Based on the visual embedding vector, the classification score region of the novel class is selected through a second preset learning method to obtain the feature construction category of the novel class.

[0021] According to one embodiment of this application, generating a classification result for the target to be detected based on the alignment result of the visual embedding vector and the text embedding vector and the similarity result between the region of interest and the feature-constructed category includes:

[0022] The visual embedding vector and the text embedding vector, as well as the region of interest and the feature construction category, are iteratively learned using an exponential moving average method to obtain the alignment results of the visual embedding vector and the text embedding vector, and the similarity results of the region of interest and the feature construction category, respectively.

[0023] The alignment result and the similarity result are weighted and calculated to generate the classification result of the target to be detected.

[0024] According to the zero-shot target detection method based on a visual-language pre-trained model and category prototypes in this application, after extracting the region of interest (ROI) of the target to be detected, region features are extracted from the ROI and input into a preset visual pre-trained model to obtain a visual embedding vector. Text information of the base class and novel class is input into a preset text pre-trained model to obtain a text embedding vector. Based on the visual embedding vector, a feature-constructed category is obtained according to the classification scores of the base class and novel class. The classification result of the target to be detected is generated based on the alignment result of the visual embedding vector and the text embedding vector, and the similarity result between the ROI and the feature-constructed category. This solves the problems of high dataset annotation costs, high resource consumption, long processing time, and lack of optimization methods for the visual-language pre-trained model during pre-training, which reduce the accuracy of zero-shot target detection. By introducing the concept of feature-constructed categories into zero-shot target detection, the feature-constructed categories of known and unknown categories are continuously optimized during training, fully leveraging the advantages of the visual-language pre-trained model and improving the progress of zero-shot target detection.

[0025] A second aspect of this application provides a zero-shot detection device based on a vision-language pre-trained model and a category prototype, comprising:

[0026] An extraction module is used to acquire the target to be detected, extract the region of interest of the target to be detected, and extract volume-level region features from the region of interest;

[0027] The input module is used to input the object-level region features into a preset visual pre-trained model to obtain visual embedding vectors, and to input the text information of the base class and novel class into a preset text pre-trained model to obtain text embedding vectors; and

[0028] The generation module is used to construct a feature category based on the visual embedding vector, according to the classification score region of the base class and the classification score region of the novel class, and generate a classification result of the target to be detected based on the alignment result of the visual embedding vector and the text embedding vector and the similarity result between the region of interest and the feature-constructed category.

[0029] According to one embodiment of this application, the extraction module is specifically used for:

[0030] Based on the target to be detected, visual weight information in the target to be detected is obtained to obtain the visual feature value of the target to be detected;

[0031] The region of interest (ROI) of the target to be detected is extracted by the RPN based on the visual feature values, and object-level region features that meet preset irrelevance conditions are extracted by multiple cascaded RoIAlign modules.

[0032] According to one embodiment of this application, before inputting the text information of the base class and the novel class into a preset text pre-trained model to obtain text embedding vectors, the input module is further configured to:

[0033] Based on the target to be detected, the text weight information in the target to be detected is obtained, and the text feature value of the target to be detected is obtained.

[0034] The text information of the base class and the novel class is extracted based on the text feature values.

[0035] According to one embodiment of this application, the generation module is specifically used for:

[0036] Based on the visual embedding vector, the classification score region of the base class is selected through a first preset learning method to obtain the feature construction category of the base class;

[0037] Based on the visual embedding vector, the classification score region of the novel class is selected through a second preset learning method to obtain the feature construction category of the novel class.

[0038] According to one embodiment of this application, the generation module is specifically used for:

[0039] The visual embedding vector and the text embedding vector, as well as the region of interest and the feature construction category, are iteratively learned using an exponential moving average method to obtain the alignment results of the visual embedding vector and the text embedding vector, and the similarity results of the region of interest and the feature construction category, respectively.

[0040] The alignment result and the similarity result are weighted and calculated to generate the classification result of the target to be detected.

[0041] According to the zero-shot detection device based on a visual-language pre-trained model and category prototypes in this application, after extracting the region of interest (ROI) of the target to be detected, regional features are extracted from the ROI and input into a preset visual pre-trained model to obtain a visual embedding vector. Text information of the base class and novel class is input into a preset text pre-trained model to obtain a text embedding vector. Based on the visual embedding vector, a feature-constructed category is obtained according to the classification scores of the base class and novel class. The classification result of the target to be detected is generated based on the alignment result of the visual embedding vector and the text embedding vector, and the similarity result between the ROI and the feature-constructed category. This solves the problems of high dataset annotation costs, high resource consumption, long processing time, and lack of optimization methods for the visual-language pre-trained model during pre-training, which reduce the accuracy of zero-shot target detection. By introducing the concept of feature-constructed categories into zero-shot target detection, the feature-constructed categories of known and unknown categories are continuously optimized during training, fully leveraging the advantages of the visual-language pre-trained model and improving the progress of zero-shot target detection.

[0042] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the zero-shot detection method based on a vision-language pre-trained model and category prototype as described in the above embodiments.

[0043] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the zero-shot detection method based on a vision-language pre-trained model and a category prototype as described in the above embodiments.

[0044] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0045] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0046] Figure 1 is a flowchart of a zero-shot detection method based on a vision-language pre-trained model and a category prototype according to an embodiment of this application;

[0047] Figure 2 is a pre-training schematic diagram according to an embodiment of this application;

[0048] Figure 3 is a block diagram of a zero-shot detection device based on a vision-language pre-trained model and a category prototype according to an embodiment of this application;

[0049] Figure 4 is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0050] Embodiments of the present invention are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0051] The following describes a zero-shot detection method based on a visual-language pre-trained model and category prototype according to an embodiment of the present invention, with reference to the accompanying drawings. Addressing the problems mentioned in the background art, such as high dataset annotation costs, high resource consumption, long processing time, and lack of optimization methods for the visual-language pre-trained model, which reduce the accuracy of zero-shot target detection, this application provides a zero-shot detection method based on a visual-language pre-trained model and category prototype. In this method, after extracting the region of interest (ROI) of the target to be detected, regional features are extracted from the ROI and input into a preset visual pre-trained model to obtain a visual embedding vector. The text information of the base class and novel class is input into a preset text pre-trained model to obtain a text embedding vector. Based on the visual embedding vector, a feature-constructed category is obtained according to the classification score regions of the base class and novel class. Finally, a classification result of the target to be detected is generated based on the alignment result of the visual embedding vector and the text embedding vector, and the similarity result between the ROI and the feature-constructed category. This solves the problems of high dataset annotation costs, high resource consumption, long training time, and lack of optimization methods for vision-language pre-trained models during pre-training, which reduce the accuracy of zero-shot object detection. By introducing the concept of feature construction category in zero-shot object detection, the system continuously optimizes the feature construction category of known and unknown categories during training, giving full play to the advantages of vision-language pre-trained models and improving the progress of zero-shot object detection.

[0052] Specifically, Figure 1 is a flowchart illustrating a zero-shot detection method based on a vision-language pre-trained model and a category prototype provided in an embodiment of this application.

[0053] As shown in Figure 1, this zero-shot detection method based on a vision-language pre-trained model and category prototype includes the following steps:

[0054] In step S101, the target to be detected is obtained, and the region of interest of the target to be detected is extracted. Volume-level region features are extracted from the region of interest.

[0055] Furthermore, in some embodiments, the region of interest (ROI) of the target to be detected is extracted, and object-level region features are extracted from the ROI. This includes: obtaining visual weight information in the target to be detected based on the target to be detected, and obtaining visual feature values ​​of the target to be detected; extracting the ROI of the target to be detected based on the visual feature values ​​through the RPN, and extracting object-level region features that meet preset irrelevance conditions through multiple cascaded RoIAlign modules.

[0056] Among them, the preset irrelevant conditions can be relevant conditions set by those skilled in the art according to the detection requirements, and are not specifically limited here.

[0057] Specifically, in the embodiments of this application, when performing zero-sample detection on the target to be detected, the target to be detected is first obtained through the backbone network; secondly, after obtaining the target to be detected, the visual weight information in the target to be detected is obtained by fixing the Visual Encoder, thereby obtaining the visual feature value of the target to be detected. At the same time, the Region of Interest (ROI) of the target to be detected is extracted based on the visual feature value using the RPN, and the object-level region features that meet the preset irrelevance conditions are extracted through multiple cascaded RoIAlign modules, so as to improve the quality of the object-level region features.

[0058] In step S102, the object-level region features are input into a preset visual pre-training model to obtain a visual embedding vector, and the text information of the base class and novel class is input into a preset text pre-training model to obtain a text embedding vector.

[0059] Furthermore, in some embodiments, before inputting the text information of the base class and novel class into a preset text pre-trained model to obtain the text embedding vector, the method further includes: obtaining the text weight information in the target to be detected based on the target to be detected, and obtaining the text feature value of the target to be detected; and extracting the text information of the base class and novel class based on the text feature value.

[0060] The preset visual pre-training model and the preset text pre-training model can be relevant training models set by those skilled in the art according to detection requirements, and are not specifically limited here.

[0061] Specifically, as shown in Figure 2, after extracting the object-level region features that meet the preset irrelevant conditions, the embodiments of this application input the object-level region features into a preset visual pre-training model, such as CLIPVisual Encoder, to obtain the corresponding visual embedding vector.

[0062] Furthermore, in this embodiment, based on the target to be detected, the text weight information in the target to be detected is obtained by fixing the Text Encoder, thereby obtaining the text feature value of the target to be detected. The text information of the base class and novel class is extracted based on the text feature value. Then, the extracted text information of the base class and novel class can be input into a preset text pre-training model, such as CLIP's Text Encoder, to obtain the text embedding vector of the base class and the text embedding vector of the novel class. Thus, this embodiment of the application obtains the text feature value of the novel class without increasing the additional training and annotation costs.

[0063] In step S103, based on the visual embedding vector, the feature construction category is obtained according to the classification score region of the base class and the classification score region of the novel class, and the classification result of the target to be detected is generated according to the alignment result of the visual embedding vector and the text embedding vector and the similarity result between the region of interest and the feature construction category.

[0064] Furthermore, in some embodiments, the feature construction category is obtained based on the classification score region of the base class and the classification score region of the novel class, including: selecting the classification score region of the base class based on the visual embedding vector using a first preset learning method to obtain the feature construction category of the base class; and selecting the classification score region of the novel class based on the visual embedding vector using a second preset learning method to obtain the feature construction category of the novel class.

[0065] Furthermore, in some embodiments, the classification result of the target to be detected is generated based on the alignment result of the visual embedding vector and the text embedding vector and the similarity result of the region of interest and the feature construction category. This includes: iteratively learning the visual embedding vector and the text embedding vector and the region of interest and the feature construction category using an exponential moving average method to obtain the alignment result of the visual embedding vector and the text embedding vector, and the similarity result of the region of interest and the feature construction category, respectively; and weighting the alignment result and the similarity result to generate the classification result of the target to be detected.

[0066] Specifically, as shown in Figure 2, in this embodiment of the application, based on the obtained visual embedding vector, firstly, in the baseclass, the classification score region of the baseclass is selected using a first preset learning method, for example, by selecting the region with a classification score of Top-K in the baseclass through ground truth, thereby obtaining the feature construction category Prototype of the baseclass.

[0067] Secondly, in the novel class, the classification score region of the novel class is selected by the second preset learning method. For example, the region with the classification score of Top-K in the novel class is selected by pseudo label, so as to obtain the feature construction category prototype of the novel class. Thus, the embodiments of this application use the rich semantic mining capabilities of the preset visual pre-trained model and the preset text pre-trained model to obtain a set of prototypes that can represent the typical features of the corresponding category region.

[0068] Finally, the obtained feature construction categories Prototype of the base class, Prototype of the novel class, the region of interest of the target to be detected, as well as the visual embedding vector and the text embedding vector, are iteratively learned using the exponential moving average method to obtain the alignment results of the visual embedding vector and the text embedding vector, as well as the similarity results between the region of interest and the feature construction category. Then, the alignment results and similarity results are weighted and calculated to generate the classification result of the target to be detected. This improves the zero-shot detection capability of the pre-trained visual model and the pre-trained text model in the novel class by introducing more prior knowledge, while also ensuring the detection accuracy in the base class.

[0069] In summary, through the discussion and analysis of the above embodiments, this application can bring the following beneficial effects:

[0070] (1) A method for zero-shot target detection based on the visual-language hybrid model VL-PLM in related technologies. In this application embodiment, the pseudo-label generation and detector training in the VL-PLM model are merged into one stage, thereby simplifying the training process of the model and enabling end-to-end training of the model.

[0071] (2) While ensuring that the detection accuracy of the base class remains unchanged, this application embodiment introduces the idea of ​​category prototype design into the zero-shot target detection task, creatively designs a set of category prototype generation methods for the base class and novel class, and continuously optimizes them during the training process to give full play to the inherent advantages of the vision-language pre-trained model and achieve higher accuracy zero-shot target detection. Among them, this application embodiment expects to improve the AP (Average Precision) value of the novel class by about 3% compared with the VL-PLM model (that is, to reach about 37%), thereby improving the detection performance of the existing mainstream methods.

[0072] According to the zero-shot target detection method based on a visual-language pre-trained model and category prototypes in this application, after extracting the region of interest (ROI) of the target to be detected, region features are extracted from the ROI and input into a preset visual pre-trained model to obtain a visual embedding vector. Text information of the base class and novel class is input into a preset text pre-trained model to obtain a text embedding vector. Based on the visual embedding vector, a feature-constructed category is obtained according to the classification scores of the base class and novel class. The classification result of the target to be detected is generated based on the alignment result of the visual embedding vector and the text embedding vector, and the similarity result between the ROI and the feature-constructed category. This solves the problems of high dataset annotation costs, high resource consumption, long processing time, and lack of optimization methods for the visual-language pre-trained model during pre-training, which reduce the accuracy of zero-shot target detection. By introducing the concept of feature-constructed categories into zero-shot target detection, the feature-constructed categories of known and unknown categories are continuously optimized during training, fully leveraging the advantages of the visual-language pre-trained model and improving the progress of zero-shot target detection.

[0073] Figure 3 is a block diagram of a zero-shot detection device based on a vision-language pre-trained model and a category prototype according to an embodiment of this application.

[0074] As shown in Figure 3, the zero-shot detection device 10 based on the vision-language pre-trained model and category prototype includes: an extraction module 100, an input module 200, and a generation module 300.

[0075] The extraction module 100 is used to acquire the target to be detected, extract the region of interest of the target to be detected, and extract volume-level region features from the region of interest.

[0076] Input module 200 is used to input object-level region features into a preset visual pre-trained model to obtain visual embedding vectors, and to input base class and novel class text information into a preset text pre-trained model to obtain text embedding vectors; and

[0077] The generation module 300 is used to obtain feature-constructed categories based on visual embedding vectors, according to the classification score regions of the base class and the classification score regions of the novel class, and to generate classification results of the target to be detected based on the alignment results of the visual embedding vectors and text embedding vectors and the similarity results between the regions of interest and the feature-constructed categories.

[0078] Furthermore, in some embodiments, the extraction module 100 is specifically used for:

[0079] Based on the target to be detected, the visual weight information of the target to be detected is obtained, and the visual feature value of the target to be detected is obtained.

[0080] The RPN extracts the region of interest of the target to be detected based on visual feature values, and extracts the object-level region features that meet the preset irrelevance conditions through multiple cascaded RoIAlign modules.

[0081] Furthermore, in some embodiments, before inputting the text information of the base class and novel class into a preset text pre-trained model to obtain the text embedding vector, the input module 200 is also used for:

[0082] Based on the target to be detected, the text weight information in the target to be detected is obtained, and the text feature value of the target to be detected is obtained.

[0083] Extract text information of base class and novel class based on text feature values.

[0084] Furthermore, in some embodiments, the generation module 300 is specifically used for:

[0085] Based on the visual embedding vector, the classification score region of the base class is selected through the first preset learning method to obtain the feature construction category of the base class;

[0086] Based on the visual embedding vector, the classification score region of the novel class is selected through the second preset learning method to obtain the feature construction category of the novel class.

[0087] Furthermore, in some embodiments, the generation module 300 is specifically used for:

[0088] The visual embedding vector and text embedding vector, as well as the region of interest and the feature construction category, are iteratively learned using the exponential moving average method to obtain the alignment results of the visual embedding vector and text embedding vector, and the similarity results of the region of interest and the feature construction category, respectively.

[0089] The alignment and similarity results are weighted and calculated to generate the classification result of the target to be detected.

[0090] According to the zero-shot detection device based on a visual-language pre-trained model and category prototypes in this application, after extracting the region of interest (ROI) of the target to be detected, regional features are extracted from the ROI and input into a preset visual pre-trained model to obtain a visual embedding vector. Text information of the base class and novel class is input into a preset text pre-trained model to obtain a text embedding vector. Based on the visual embedding vector, a feature-constructed category is obtained according to the classification scores of the base class and novel class. The classification result of the target to be detected is generated based on the alignment result of the visual embedding vector and the text embedding vector, and the similarity result between the ROI and the feature-constructed category. This solves the problems of high dataset annotation costs, high resource consumption, long processing time, and lack of optimization methods for the visual-language pre-trained model during pre-training, which reduce the accuracy of zero-shot target detection. By introducing the concept of feature-constructed categories into zero-shot target detection, the feature-constructed categories of known and unknown categories are continuously optimized during training, fully leveraging the advantages of the visual-language pre-trained model and improving the progress of zero-shot target detection.

[0091] Figure 4 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0092] The memory 401, the processor 402, and the computer program stored on the memory 401 and capable of running on the processor 402.

[0093] When the processor 402 executes the program, it implements the zero-shot detection method based on the vision-language pre-trained model and category prototype provided in the above embodiments.

[0094] Furthermore, electronic devices also include:

[0095] Communication interface 403 is used for communication between memory 401 and processor 402.

[0096] The memory 401 is used to store computer programs that can run on the processor 402.

[0097] The memory 401 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0098] If the memory 401, processor 402, and communication interface 403 are implemented independently, they can be interconnected via a bus to communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one thick line is used in Figure 4, but this does not indicate that there is only one bus or one type of bus.

[0099] Optionally, in a specific implementation, if the memory 401, processor 402, and communication interface 403 are integrated on a single chip, then the memory 401, processor 402, and communication interface 403 can communicate with each other through an internal interface.

[0100] Processor 402 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0101] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the zero-shot detection method based on a vision-language pre-trained model and a category prototype as described above.

[0102] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0103] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0104] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A zero-shot detection method based on a vision-language pre-trained model and a category prototype, characterized in that, The process includes the following steps: acquiring the target to be detected, extracting the region of interest of the target to be detected, and extracting volume-level region features from the region of interest; The object-level region features are input into a preset visual pre-training model to obtain a visual embedding vector, and the text information of the base class and novel class is input into a preset text pre-training model to obtain a text embedding vector; and based on the visual embedding vector, a feature-constructed category is obtained according to the classification score region of the base class and the classification score region of the novel class, and a classification result of the target to be detected is generated according to the alignment result of the visual embedding vector and the text embedding vector and the similarity result between the region of interest and the feature-constructed category; Extracting the region of interest of the target to be detected and extracting body-level region features from the region of interest includes: obtaining visual weight information in the target to be detected based on the target to be detected, and obtaining the visual feature value of the target to be detected; The Region Generation Network (RPN) extracts the region of interest (ROI) of the target to be detected based on the visual feature values, and extracts object-level region features that meet preset irrelevance conditions through multiple cascaded RoIAlign modules.

2. The method according to claim 1, characterized in that, Before inputting the text information of the base class and novel class into the preset text pre-training model to obtain the text embedding vector, the method further includes: obtaining the text weight information of the target to be detected based on the target to be detected, and obtaining the text feature value of the target to be detected; and extracting the text information of the base class and the novel class according to the text feature value.

3. The method according to claim 1, characterized in that, The feature construction category is obtained based on the classification score region of the base class and the classification score region of the novel class, including: selecting the classification score region of the base class based on the visual embedding vector using a first preset learning method to obtain the feature construction category of the base class; and selecting the classification score region of the novel class based on the visual embedding vector using a second preset learning method to obtain the feature construction category of the novel class.

4. The method according to claim 3, characterized in that, The step of generating a classification result for the target to be detected based on the alignment result of the visual embedding vector and the text embedding vector and the similarity result of the region of interest and the feature construction category includes: iteratively learning the visual embedding vector and the text embedding vector, as well as the region of interest and the feature construction category, using an exponential moving average method to obtain the alignment result of the visual embedding vector and the text embedding vector, and the similarity result of the region of interest and the feature construction category, respectively; and weighting the alignment result and the similarity result to generate a classification result for the target to be detected.

5. A zero-shot detection device based on a vision-language pre-trained model and a category prototype, characterized in that, It includes an extraction module for acquiring the target to be detected, extracting the region of interest of the target to be detected, and extracting volume-level region features from the region of interest; The input module is used to input the object-level region features into a preset visual pre-trained model to obtain a visual embedding vector, and input the text information of the base class and novel class into a preset text pre-trained model to obtain a text embedding vector; and the generation module is used to construct a category based on the visual embedding vector, according to the classification score region of the base class and the classification score region of the novel class, and generate a classification result of the target to be detected based on the alignment result of the visual embedding vector and the text embedding vector and the similarity result between the region of interest and the category constructed by the features; the extraction module is specifically used to: obtain the visual weight information in the target to be detected to obtain the visual feature value of the target to be detected; extract the region of interest of the target to be detected based on the visual feature value through RPN, and extract the object-level region features that meet the preset irrelevance conditions through multiple cascaded RoIAlign modules.

6. The apparatus according to claim 5, characterized in that, Before inputting the text information of the base class and novel class into the preset text pre-trained model to obtain the text embedding vector, the input module is further configured to: obtain the text weight information in the target to be detected based on the target to be detected, and obtain the text feature value of the target to be detected; and extract the text information of the base class and the novel class according to the text feature value.

7. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, the processor executing the program to implement the zero-shot detection method based on a vision-language pre-trained model and a category prototype as described in any one of claims 1-4.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the zero-shot detection method based on a vision-language pre-trained model and a category prototype as described in any one of claims 1-4.

Images