A Language Model-Driven Zero-Shot Object Detection Method and System
By acquiring external knowledge from a large language model and combining generative adversarial networks and supervised detection models, the problem of semantic confusion in zero-shot object detection is solved, and effective detection of visible and invisible classes is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING UNIV OF TECH
- Filing Date
- 2023-09-11
- Publication Date
- 2026-05-26
AI Technical Summary
Existing zero-shot object detection methods have limited feature information and restricted transferability when detecting visible and invisible classes, which makes the classifier prone to semantic confusion and unable to effectively distinguish between different categories.
By introducing a large language model to acquire external knowledge, rich semantic visual features are generated. Generative adversarial networks and supervised detection models are used for comparative learning. Combined with the concept of cycle consistency, the visual space and semantic space are aligned to generate clear visual feature boundaries.
It improves the semantic confusion problem in zero-shot object detection, enhances the discriminative power and semantic alignment of visual features, and achieves effective detection of both visible and invisible classes.
Smart Images
Figure CN117195911B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of target detection technology, and in particular to a language model-driven zero-shot target detection method and a language model-driven zero-shot target detection system. Background Technology
[0002] Object detection has long been one of the core tasks of computing. It utilizes a large amount of labeled data, and thanks to the advent of neural networks, conventional object detection has achieved remarkable results. However, in real-world scenarios, many categories lack sufficient data, and some categories simply lack available data altogether. Conventional object detection performs poorly on tasks with extremely uneven data distribution. Therefore, many scholars have conducted research on this issue.
[0003] Zero-shot object detection (ZSD) has attracted significant attention in current research as a method that uses auxiliary information such as semantics to transfer knowledge of visible categories to invisible categories. Related methods are mainly divided into embedding-based methods and generative model-based methods. Embedding-based methods map visual and semantic features to a common space, while generative methods utilize pre-trained generative models to generate pseudo-visual features for invisible categories.
[0004] However, in the Generalized Zero-Shot Object Detection (GZSD) task, which simultaneously detects both visible and invisible classes, the transferability of both categories is limited. This limitation stems primarily from two aspects: firstly, existing methods extract or generate semantic features based solely on labeled data, resulting in limited information contained in the features and even poor feature discriminability; secondly, there is the bias problem, because the classifier is trained using only samples and labels from the visible class and has never encountered the invisible class. Therefore, during inference, the classifier often misclassifies the invisible class as the visible class or the background class. Summary of the Invention
[0005] To address the aforementioned problems, this invention provides a language model-driven zero-shot object detection method and system. By introducing external knowledge from a large-scale language model, semantically rich textual information is obtained, which can assist in generating visual features with rich semantics and discriminative power. The contrastive learning of the supervised detection model makes the boundaries between visual features of different categories clearer. Combined with the concept of cycle consistency, the generative model's ability to understand and express visual content is further improved, enabling better alignment between the visual space and the semantic space, thus solving the problem of semantic confusion in zero-shot object detection.
[0006] To achieve the above objectives, this invention provides a language model-driven zero-shot object detection method, comprising:
[0007] A supervised detection model is trained using visible class data from the dataset;
[0008] Based on the data class names in the dataset, semantic vectors associated with the data class names are extracted from a pre-set large-scale language model to generate external knowledge;
[0009] Using the supervised detection model, visual features are extracted from the visible class images in the dataset. The extracted visual features are used as real features, and pseudo-visual features of the visible class are synthesized based on the external knowledge. Generative adversarial training is then performed to obtain a generative adversarial network.
[0010] The invisible class pseudo-visual features in the dataset are synthesized using the generative adversarial network, and an invisible visual feature classifier is trained using the invisible class pseudo-visual features and the corresponding data category labels.
[0011] Based on the invisible visual feature classifier, the parameters of the classifier in the supervised detection model are fused to obtain the updated supervised detection model;
[0012] The image data to be detected is input into the supervised detection model, and the target detection result is output.
[0013] In the above technical solution, preferably, the specific process of training a supervised detection model using visible class data in the dataset includes:
[0014] The data in the dataset is divided into visible data and invisible data;
[0015] Using the visible class data, a two-stage detection model Faster RCNN is trained in a supervised manner as the supervised detection model.
[0016] In the above technical solution, preferably, the specific process of extracting semantic vectors associated with the data class names from a preset large-scale language model based on the data class names in the dataset to generate external knowledge includes:
[0017] Based on prompt learning, the data class names in the dataset are substituted into the prompt template, and the text description associated with the data class names is captured by the large language model;
[0018] Generate a relation description based on the text description, and input the text description and the relation description into a text encoding model to obtain the semantic vector;
[0019] Calculate the similarity between the semantic vector and the data class name, and accumulate the semantic vector using the similarity as a weight to obtain the external knowledge.
[0020] In the above technical solution, preferably, the specific process of using the extracted visual features as real features, synthesizing visible pseudo-visual features based on the external knowledge, and performing generative adversarial training to obtain a generative adversarial network includes:
[0021] The external knowledge is concatenated with Gaussian noise and input into the generator to obtain the visible pseudo-visual features;
[0022] Based on the WGAN model, the real features and the visible pseudo-visual features extracted by the supervised detection model are input into the discriminator for generative adversarial training.
[0023] The generative adversarial network training is complete when the preset loss function is reached.
[0024] In the above technical solution, preferably, the process of synthesizing the visible pseudo-visual features based on the external knowledge further includes:
[0025] Features generated from the same class of data in the dataset are used as positive examples, and features generated from different classes of data are used as negative examples. The contrast loss is calculated based on the positive and negative examples.
[0026] Based on the contrast loss, the visual feature distributions of data of the same type are made closer together, while the visual feature distributions of data of different types are made farther apart.
[0027] A linear mapping layer is used to map the generated features from the visual space to the original semantic space for alignment, and the mean square error corresponding to the original semantics is calculated to ensure that the generated features retain the same semantics as the corresponding labels.
[0028] In the above technical solution, preferably, the specific process of using the generative adversarial network to synthesize invisible class pseudo-visual features from the invisible class data in the dataset, and using the invisible class pseudo-visual features and the corresponding data category labels to train an invisible visual feature classifier includes:
[0029] The invisible class data in the dataset is input into the generative adversarial network to obtain the visual features corresponding to the invisible class data, which are used as the invisible class visual features.
[0030] An invisible visual feature classifier is trained by using the invisible visual features as input and the corresponding data class labels as output.
[0031] This invention also proposes a language model-driven zero-shot object detection system, characterized in that it applies the language model-driven zero-shot object detection method disclosed in any of the above technical solutions, including:
[0032] The detection model training module is used to train a supervised detection model using visible class data in the dataset.
[0033] The external knowledge introduction module is used to extract semantic vectors associated with the data class names from a preset large language model based on the data class names in the dataset, and generate external knowledge.
[0034] The generative model training module is used to extract visual features from visible class images in the dataset using the supervised detection model, and use the extracted visual features as real features, synthesize visible class pseudo-visual features based on the external knowledge, and perform generative adversarial training to obtain a generative adversarial network.
[0035] The invisible class training module is used to synthesize invisible class pseudo-visual features from the invisible class data in the dataset using the generative adversarial network, and to train an invisible visual feature classifier using the invisible class pseudo-visual features and the corresponding data category labels.
[0036] The classification model fusion module is used to fuse the parameters of the classifier in the supervised detection model according to the invisible visual feature classifier to obtain an updated supervised detection model.
[0037] The zero-shot target detection module is used to input the image data of the target to be detected into the supervised detection model and output the target detection result.
[0038] In the above technical solution, preferably, the external knowledge introduction module is specifically used for:
[0039] Based on prompt learning, the data class names in the dataset are substituted into the prompt template, and the text description associated with the data class names is captured by the large language model;
[0040] Generate a relation description based on the text description, and input the text description and the relation description into a text encoding model to obtain the semantic vector;
[0041] Calculate the similarity between the semantic vector and the data class name, and accumulate the semantic vector using the similarity as a weight to obtain the external knowledge.
[0042] In the above technical solution, preferably, the generative model training module is specifically used for:
[0043] The external knowledge is concatenated with Gaussian noise and input into the generator to obtain the visible pseudo-visual features;
[0044] Based on the WGAN model, the real features and the visible pseudo-visual features extracted by the supervised detection model are input into the discriminator for generative adversarial training.
[0045] The generative adversarial network training is complete when the preset loss function is reached.
[0046] In the above technical solution, preferably, the invisible class training module is specifically used for:
[0047] The invisible class data in the dataset is input into the generative adversarial network to obtain the visual features corresponding to the invisible class data, which are used as the invisible class visual features.
[0048] An invisible visual feature classifier is trained by using the invisible visual features as input and the corresponding data class labels as output.
[0049] Compared with existing technologies, the beneficial effects of this invention are as follows: by introducing external knowledge from large-scale language models, semantically rich text information is obtained, which assists in generating visual features with rich semantics and discriminativeness. The contrastive learning of supervised detection models makes the boundaries between visual features of different categories clearer. Combined with the idea of cycle consistency, the generative model's ability to understand and express visual content is further improved, making the visual space and semantic space better aligned, thus solving the problem of semantic confusion in zero-shot object detection. Attached Figure Description
[0050] Figure 1 This is a flowchart illustrating a language model-driven zero-shot target detection method disclosed in an embodiment of the present invention.
[0051] Figure 2 This is a schematic diagram of the architecture of a language model-driven zero-shot object detection method disclosed in an embodiment of the present invention;
[0052] Figure 3 This is a schematic diagram of a process for obtaining external knowledge from a large language model, as disclosed in one embodiment of the present invention.
[0053] Figure 4 This is a schematic diagram of a language model-driven zero-shot target detection system disclosed in an embodiment of the present invention.
[0054] In the diagram, the correspondence between the components and the reference numerals is as follows:
[0055] 1. Detection model training module, 2. External knowledge introduction module, 3. Generative model training module, 4. Invisible class training module, 5. Classification model fusion module, 6. Zero-shot object detection module. Detailed Implementation
[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0057] The present invention will now be described in further detail with reference to the accompanying drawings:
[0058] like Figure 1 As shown, a language model-driven zero-shot target detection method according to the present invention includes:
[0059] A supervised detection model is trained using visible class data from the dataset;
[0060] Based on the data class names in the dataset, semantic vectors associated with the data class names are extracted from a pre-set large-scale language model to generate external knowledge;
[0061] Using a supervised detection model, visual features are extracted from visible images in the dataset. The extracted visual features are used as real features, and pseudo-visual features of the visible class are synthesized based on external knowledge. Generative adversarial training is then performed to obtain a generative adversarial network.
[0062] Generative adversarial networks are used to synthesize invisible class pseudo-visual features from the invisible class data in the dataset, and invisible class pseudo-visual features and corresponding data category labels are used to train an invisible visual feature classifier;
[0063] Based on the invisible visual feature classifier, the parameters of the classifier in the supervised detection model are fused to obtain the updated supervised detection model;
[0064] The image data to be detected is input into the supervised detection model, and the target detection result is output.
[0065] In this implementation, by introducing external knowledge from a large language model, semantically rich textual information is obtained, which helps to generate visual features with rich semantics and discriminativeness. The contrastive learning of the supervised detection model makes the boundaries between visual features of different categories clearer. Combined with the idea of cycle consistency, the generative model's ability to understand and express visual content is further improved, so that the visual space and semantic space are better aligned, thus solving the problem of semantic confusion in zero-shot object detection.
[0066] like Figure 2As shown, specifically, a traditional supervised detection model is first trained in a fully supervised manner using visible class data in the dataset to achieve target detection on visible class data.
[0067] Secondly, the mainstream zero-shot object detection methods based on generative methods in the existing technology mainly use class names as semantic vectors to generate pseudo-visual features for invisible classes. However, using only class names cannot provide enough semantic information. Therefore, this invention uses external knowledge obtained from large language models to assist in generating visual features. The resulting visual features can contain more semantic information, thereby improving the discriminability of visual features and increasing the degree of alignment between semantic features and visual features. Thus, it can improve the zero-shot object detection capability of the model.
[0068] Then, the visual features are extracted from the visible class images using the supervised detection model described above. Using these visual features and the external knowledge of the corresponding categories obtained from the large language model, a generative adversarial network is obtained through generative adversarial training.
[0069] Since invisible classes only have auxiliary information such as semantic vectors and lack visual features, it is impossible to train a classifier for invisible class visual images. Therefore, based on this, the aforementioned generative adversarial network is used to synthesize pseudo-visual features for invisible class data. Using these pseudo-visual features and their category labels, an additional invisible visual feature classifier is trained in a fully supervised manner to classify invisible visual features.
[0070] Finally, to handle the more challenging generalized zero-shot object detection, the parameters of the classifier for invisible visual features and the classifier for the supervised detection model that classifies visible visual features are fused; that is, the parameters of the classifier in the supervised detection model are updated. Thus, this supervised detection model can detect visual features in both visible and invisible data.
[0071] In the above embodiments, preferably, the specific process of training a supervised detection model using visible class data in the dataset includes:
[0072] Divide the data in the dataset into visible data and invisible data;
[0073] We use visible class data to train a two-stage detection model, Faster RCNN, in a supervised manner as a supervised detection model.
[0074] Among existing technologies, mainstream detection models are mainly divided into single-stage and two-stage models. This invention selects the two-stage Faster R-CNN detection model, which mainly includes a backbone network, a region proposal network, a region of interest pooling layer, a classification network, and a regression network. After the supervised detection model is trained, its backbone network can extract relatively good visual features from visual images that are distributed in the same way as the training data. The classification network serves as the classifier for object detection in visible images disclosed in the above embodiments.
[0075] like Figure 3 As shown, in the above implementation, preferably, the specific process of extracting semantic vectors associated with data class names from a preset large-scale language model based on data class names in the dataset to generate external knowledge includes:
[0076] Based on cue learning, the data class names in the dataset are substituted into the cue template, and the text descriptions associated with the data class names are captured by a large language model;
[0077] Generate relational descriptions from text descriptions, and input the text descriptions and relational descriptions into a text encoding model to obtain semantic vectors;
[0078] Calculate the similarity between semantic vectors and data class names, and accumulate the semantic vectors using similarity as weights to obtain external knowledge.
[0079] External knowledge refers to relevant textual information mined from large-scale text data using Large Language Models (LLMs), including attribute hints and relation hints. Specifically, this invention first uses attribute hints to extract category-related attribute information from the LLM; for example, the attributes corresponding to the "dog" category include "coat color," "size," and "breed." Then, relation hints are used to establish semantic relationships between the extracted attributes and categories from the LLM; for example, the relationship between the "dog" category and the "coat color" attribute is "have," while the relationship between the "dog" category and the "size" attribute is "belongs to." Finally, this invention encodes these captured semantic relationships into a vector representation, called external knowledge. This external knowledge includes class attribute knowledge and knowledge of inter-class dependencies. With the assistance of external knowledge, the semantic information embedded in visual features by generative adversarial networks can be enhanced, providing the necessary conditions for subsequent alignment of the semantic space and visual space.
[0080] Specifically, inspired by cue-based learning, this invention develops two cues to capture class attribute and relationship information. This invention uses the phrase "Please describe the shape and color of..." <class>"and the scenes where it often appears." As an attribute hint template, "Please introduce the relationship between..." <prompt>and <class>The period (.) serves as a relational hint template. For each class c, this invention uses the class name w. c Replace the template <class>Then, the replaced attribute hint template is sent to a large language model to obtain a text description.
[0081] Because large language models contain a wealth of knowledge, This will likely contain specific class attributes and other related class entities. To capture the dependencies between categories, we need to identify... All nouns and randomly select k nouns from them. Then replace the relation hint template with each selected noun. <prompt>Generate relation descriptions therefore, Provide class attribute information, while Key Focus The interdependencies between them and category c are crucial for zero-shot learning.
[0082] Finally The text is fed into a text encoding model T to obtain the required semantic vector. Then, these semantic vectors are summed according to their weights in the following manner to obtain the external knowledge s. c :
[0083]
[0084] Where, λ k By calculating the semantic vectors between With class name w c The similarity between semantic codes is obtained as follows:
[0085]
[0086] In the above embodiments, preferably, the specific process of using the extracted visual features as real features, synthesizing visible pseudo-visual features based on external knowledge, and performing generative adversarial training to obtain a generative adversarial network includes:
[0087] External knowledge is concatenated with Gaussian noise and input into the generator to obtain visible pseudo-visual features;
[0088] Based on the WGAN model, the real features and visible pseudo-visual features extracted by the supervised detection model are input into the discriminator for generative adversarial training;
[0089] The generative adversarial network training is complete when the preset loss function is reached.
[0090] In this implementation, the supervised detection model disclosed in the above-described implementation is first used to save the foreground and background region visual features of the images in the dataset, after passing through backbone, RPN, and ROI pooling layers, to a binary file with uniform dimensions. Simultaneously, the label information corresponding to these visual features is also saved to another binary file in the order of the visual features. When the model needs to be trained using this data later, it can be directly read from the file without having to repeat the process each time.
[0091] After expanding the semantic information, a feature generator is trained using semantic vectors incorporating external knowledge. Specifically, s c The conditional information is input into a traditional generative adversarial network. For class c in the visible class, the corresponding external knowledge s is... c It is connected to random noise z and input into generator G to obtain the corresponding pseudo-visual features. Features of class c output by the trained Faster R-CNN As a true characteristic, and The input is fed into the discriminator for generative adversarial training, and the resulting generative adversarial network is obtained after training.
[0092] The loss function during generative adversarial training is as follows:
[0093]
[0094] in, These are real visual features. These are fake visual features synthesized by the generator based on semantic vectors, s c These are their corresponding semantic vectors. λ is a hyperparameter of the gradient-constrained generalization (GP).
[0095] While the WGAN model mentioned above can already match visual features and semantic vectors to a certain extent, this invention adds classification constraints during the WGAN training process to more accurately determine the correct category of the synthesized pseudo-visual features. The loss function is:
[0096] L CLS =-E[logp(y c |G(S c ,z);φ cls1 )]
[0097] Where y∈Y s Let y represent the true value, G represent the generator, and logp(y|G) represent the result of the linear classifier φ. cls1 Calculated class probability prediction. φ cls1 It is a classifier in Faster RCNN that keeps its parameters frozen while classifying synthetic pseudo-visual features.
[0098] In the above embodiments, preferably, the process of synthesizing visible pseudo-visual features based on external knowledge further includes:
[0099] Features generated from the same class of data in the dataset are used as positive examples, and features generated from different classes of data are used as negative examples. The contrast loss is calculated based on the positive and negative examples.
[0100] Based on contrast loss, the visual feature distributions of data of the same type are made closer together, while the visual feature distributions of data of different types are made farther apart.
[0101] A linear mapping layer is used to map the generated features from the visual space to the original semantic space for alignment, and the mean square error corresponding to the original semantics is calculated to ensure that the generated features retain the same semantics as the corresponding labels.
[0102] Specifically, features generated by different classes overlap in visual space. Therefore, a contrastive approach is used to make visual features of the same class more closely distributed and visual features of different classes more widely distributed. Specifically, for Features generated from the same class are considered positive examples, while features generated from different classes are considered negative examples. The contrastive loss is calculated as follows:
[0103]
[0104] in,
[0105] Where τ represents the temperature coefficient during optimization, l [k≠c] This is an indicator function, taking the value 0 if and only if k = c, and 1 otherwise. N represents the N samples input into the network from the same batch. The purpose of this loss is to shorten... The distance between it and its positive example, and simultaneously push away The distance between it and its negative examples enables better discriminative power of visual features.
[0106] Furthermore, the generated visual feature space is aligned with the original semantic space. Specifically, a linear mapping layer is used to map the generated features from the visual space back to the original semantic space, and then the mean squared error corresponding to the original semantics is calculated.
[0107]
[0108] L CYC Ensuring that the generated features retain the same semantics as their real labels will bring about cycle consistency and improve the quality of the generated visual features.
[0109] In the above embodiments, preferably, the specific process of using a generative adversarial network to synthesize invisible class pseudo-visual features from the invisible class data in the dataset, and using the invisible class pseudo-visual features and the corresponding data category labels to train an invisible visual feature classifier includes:
[0110] The invisible class data in the dataset is input into the generative adversarial network to obtain the visual features corresponding to the invisible class data, which are used as the visual features of the invisible class.
[0111] An invisible visual feature classifier is trained by using invisible visual features as input and the corresponding data class labels as output.
[0112] In this implementation, after WGAN training is completed, the semantic vectors of the invisible classes are... The Gaussian noise is fed into the generator G to obtain pseudo-visual features corresponding to the categories of the semantic vector. Then use these pseudo-visual features Using a fully supervised approach, a new classifier φ is trained with the corresponding label c. cls2 Where φ cls2 It is a classifier consisting of a single fully connected layer, which ultimately classifies visual features of invisible classes.
[0113] For the classifier φ that has been obtained for classifying the visible class cls1 And a classifier φ for classifying invisible classes cls2 To simultaneously classify visible and invisible data, the two classifiers are merged in a specific ratio to obtain a classifier capable of classifying both visible and invisible data.
[0114] φ cls =β0φ cls1 +β1φ cls2
[0115] Where β0 and β1 are the classifiers φ cls1 and classifier φ cls2 The weights of the two classifiers. φ after the two classifiers are fused. cls It can replace the original classifier in Faster RCNN and achieve zero-shot object detection and generalized zero-shot object detection.
[0116] After the above implementation, a comparison was made with existing target detection methods, and the results are shown in Table 1 below. Analysis of the results in Table 1 shows that the method of the present invention has better detection performance in zero-sample target detection partitioning of the target detection dataset compared to existing target detection methods.
[0117] Table 1
[0118]
[0119] like Figure 4 As shown, this invention also proposes a language model-driven zero-shot target detection system, characterized in that it applies the language model-driven zero-shot target detection method disclosed in any of the above embodiments, including:
[0120] Detection model training module 1 is used to train a supervised detection model using visible class data in the dataset.
[0121] External knowledge introduction module 2 is used to extract semantic vectors associated with data class names from a pre-set large language model based on the data class names in the dataset, and generate external knowledge.
[0122] Generative model training module 3 is used to extract visual features from visible class images in the dataset using a supervised detection model, and use the extracted visual features as real features, synthesize visible class pseudo visual features based on external knowledge, and perform generative adversarial training to obtain a generative adversarial network.
[0123] The invisible class training module 4 is used to synthesize invisible class pseudo-visual features from the invisible class data in the dataset using a generative adversarial network, and to train an invisible visual feature classifier using the invisible class pseudo-visual features and the corresponding data category labels.
[0124] The classification model fusion module 5 is used to fuse the parameters of the classifiers in the supervised detection model based on the invisible visual feature classifier to obtain the updated supervised detection model.
[0125] The zero-shot target detection module 6 is used to input the image data to be detected into the supervised detection model and output the target detection results.
[0126] In this implementation, by introducing external knowledge from a large language model and embedding semantic information into visual features, visual features with rich semantics and discriminativeness are generated. The contrastive learning of the supervised detection model makes the boundaries between different categories of visual features clearer. Combined with the idea of cycle consistency, the generative model's ability to understand and express visual content is further improved, making the visual space and semantic space better aligned and solving the problem of semantic confusion in zero-shot object detection.
[0127] In the above embodiments, preferably, the external knowledge introduction module 2 is specifically used for:
[0128] Based on cue learning, the data class names in the dataset are substituted into the cue template, and the text descriptions associated with the data class names are captured by a large language model;
[0129] Generate relational descriptions from text descriptions, and input the text descriptions and relational descriptions into a text encoding model to obtain semantic vectors;
[0130] Calculate the similarity between semantic vectors and data class names, and accumulate the semantic vectors using similarity as weights to obtain external knowledge.
[0131] In the above embodiments, preferably, the model training module 3 is specifically used for:
[0132] External knowledge is concatenated with Gaussian noise and input into the generator to obtain visible pseudo-visual features;
[0133] Based on the WGAN model, the real features and visible pseudo-visual features extracted by the supervised detection model are input into the discriminator for generative adversarial training;
[0134] The generative adversarial network training is complete when the preset loss function is reached.
[0135] In the above embodiments, preferably, the invisible class training module 4 is specifically used for:
[0136] The invisible class data in the dataset is input into the generative adversarial network to obtain the visual features corresponding to the invisible class data, which are used as the visual features of the invisible class.
[0137] An invisible visual feature classifier is trained by using invisible visual features as input and the corresponding data class labels as output.
[0138] The language model-driven zero-shot object detection system disclosed in the above embodiments has modules whose implementation methods correspond to the steps in the language model-driven zero-shot object detection method disclosed in the above embodiments. In practice, the implementation methods of each module are carried out with reference to the methods described in the above embodiments, and will not be repeated here.
[0139] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.< / prompt> < / class> < / class> < / prompt> < / class>
Claims
1. A language model-driven zero-shot object detection method, characterized in that, include: A supervised detection model is trained using visible class data from the dataset; Based on the data class names in the dataset, semantic vectors associated with the data class names are extracted from a pre-set large-scale language model to generate external knowledge; Using the supervised detection model, visual features are extracted from the visible class images in the dataset. The extracted visual features are used as real features, and pseudo-visual features of the visible class are synthesized based on the external knowledge. Generative adversarial training is then performed to obtain a generative adversarial network. The invisible class pseudo-visual features in the dataset are synthesized using the generative adversarial network, and an invisible visual feature classifier is trained using the invisible class pseudo-visual features and the corresponding data category labels. Based on the invisible visual feature classifier, the parameters of the classifier in the supervised detection model are fused to obtain the updated supervised detection model; The image data to be detected is input into the supervised detection model, and the target detection result is output.
2. The language model-driven zero-shot target detection method according to claim 1, characterized in that, The specific process of training a supervised detection model using visible class data from the dataset includes: The data in the dataset is divided into visible data and invisible data; Using the visible class data, a two-stage detection model Faster RCNN is trained in a supervised manner as the supervised detection model.
3. The language model-driven zero-shot target detection method according to claim 1, characterized in that, The specific process of generating external knowledge by extracting semantic vectors associated with the data class names from a preset large-scale language model based on the data class names in the dataset includes: Based on prompt learning, the data class names in the dataset are substituted into the prompt template, and the text description associated with the data class names is captured by the large language model; Generate a relation description based on the text description, and input the text description and the relation description into a text encoding model to obtain the semantic vector; Calculate the similarity between the semantic vector and the data class name, and accumulate the semantic vector using the similarity as a weight to obtain the external knowledge.
4. The language model-driven zero-shot target detection method according to claim 2, characterized in that, The specific process of using the extracted visual features as real features, synthesizing visible pseudo-visual features based on the external knowledge, and performing generative adversarial training to obtain a generative adversarial network includes: The external knowledge is concatenated with Gaussian noise and input into the generator to obtain the visible pseudo-visual features; Based on the WGAN model, the real features and the visible pseudo-visual features extracted by the supervised detection model are input into the discriminator for generative adversarial training; The generative adversarial network training is complete when the preset loss function is reached.
5. The language model-driven zero-shot target detection method according to claim 4, characterized in that, The process of synthesizing the visible pseudo-visual features based on the external knowledge also includes: Features generated from the same class of data in the dataset are used as positive examples, and features generated from different classes of data are used as negative examples. The contrast loss is calculated based on the positive and negative examples. Based on the contrast loss, the visual feature distributions of data of the same type are made closer together, while the visual feature distributions of data of different types are made farther apart. A linear mapping layer is used to map the generated features from the visual space to the original semantic space for alignment, and the mean square error corresponding to the original semantics is calculated to ensure that the generated features retain the same semantics as the corresponding labels.
6. The language model-driven zero-shot target detection method according to claim 4, characterized in that, The specific process of using the generative adversarial network to synthesize invisible class pseudo-visual features from the invisible class data in the dataset, and using the invisible class pseudo-visual features and the corresponding data category labels to train an invisible visual feature classifier includes: The invisible class data in the dataset is input into the generative adversarial network to obtain the visual features corresponding to the invisible class data, which are used as the visual features of the invisible class. Using the invisible visual features as input and the corresponding data class labels as output, an invisible visual feature classifier is trained.
7. A language model-driven zero-shot target detection system, characterized in that, The language model-driven zero-shot object detection method as described in any one of claims 1 to 6 includes: The detection model training module is used to train a supervised detection model using visible class data in the dataset. The external knowledge introduction module is used to extract semantic vectors associated with the data class names from a preset large language model based on the data class names in the dataset, and generate external knowledge. The generative model training module is used to extract visual features from visible class images in the dataset using the supervised detection model, and use the extracted visual features as real features, synthesize visible class pseudo-visual features based on the external knowledge, and perform generative adversarial training to obtain a generative adversarial network. The invisible class training module is used to synthesize invisible class pseudo-visual features from the invisible class data in the dataset using the generative adversarial network, and to train an invisible visual feature classifier using the invisible class pseudo-visual features and the corresponding data category labels. The classification model fusion module is used to fuse the parameters of the classifier in the supervised detection model according to the invisible visual feature classifier to obtain an updated supervised detection model. The zero-shot target detection module is used to input the image data of the target to be detected into the supervised detection model and output the target detection result.
8. The language model-driven zero-shot target detection system according to claim 7, characterized in that, The external knowledge import module is specifically used for: Based on prompt learning, the data class names in the dataset are substituted into the prompt template, and the text description associated with the data class names is captured by the large language model; Generate a relation description based on the text description, and input the text description and the relation description into a text encoding model to obtain the semantic vector; Calculate the similarity between the semantic vector and the data class name, and accumulate the semantic vector using the similarity as a weight to obtain the external knowledge.
9. The language model-driven zero-shot target detection system according to claim 7, characterized in that, The generative model training module is specifically used for: The external knowledge is concatenated with Gaussian noise and input into the generator to obtain the visible pseudo-visual features; Based on the WGAN model, the real features and the visible pseudo-visual features extracted by the supervised detection model are input into the discriminator for generative adversarial training; The generative adversarial network training is complete when the preset loss function is reached.
10. The language model-driven zero-shot target detection system according to claim 7, characterized in that, The invisible class training module is specifically used for: The invisible class data in the dataset is input into the generative adversarial network to obtain the visual features corresponding to the invisible class data, which are used as the visual features of the invisible class. Using the invisible visual features as input and the corresponding data class labels as output, an invisible visual feature classifier is trained.