Small sample open set identification method and system based on multi-modal negative prototype, and medium

Abstract

The invention discloses a small sample open set recognition method and system based on a multi-modal negative prototype and a medium, and the method comprises the steps: inputting a plurality of samples in a query set of an open set small sample task into a small sample open set recognition model, and carrying out the closed set classification and open set recognition; the training process of the small sample open set recognition model comprises the following steps: obtaining a query set sample image and a support set sample, and respectively extracting support sample visual features, query sample visual features and support semantic features; inputting the support sample visual features and the support semantic features into a feature fusion network to obtain a multi-modal category prototype; carrying out attention mechanism operation on the multi-modal category prototype and the multi-modal open weight to output a plurality of multi-modal negative prototypes; calculating the similarity between each query sample visual feature and each multi-modal negative prototype, and taking the multi-modal negative prototype with the maximum similarity as the optimal negative prototype of each query sample; according to the small sample open set identification method, the small sample identification stability and accuracy are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to a small sample open set recognition method and system based on multi-modal negative prototypes and a medium. BACKGROUND

[0002] Small sample open set recognition is a research direction that has attracted widespread attention in the field of artificial intelligence and pattern recognition in recent years. It aims to enable the model to not only accurately complete the classification task of the closed set, but also effectively recognize samples from unknown categories under the condition of only a few known class samples. This technology is particularly suitable for application scenarios with high data collection costs, difficult labeling or complex deployment environments, such as medical diagnosis, autonomous driving, security monitoring, etc. In these real-world applications, the model often cannot foresee all possible categories, so the ability to handle unknown categories is particularly important. For example, in the rare disease diagnosis task, not only known diseases need to be identified based on limited samples, but also the ability to discover unknown variants or rare diseases is needed to assist doctors in making further judgments.

[0003] Small sample open set recognition combines small sample learning and open set recognition. Small sample learning focuses on how to complete the learning task of new categories through a small number of samples, emphasizing the generalization ability of the model; open set recognition aims to remain robust and recognize in front of categories that do not appear in the training set. However, both have obvious limitations: small sample learning methods usually assume that test categories and training categories come from the same set, making it difficult to handle unknown classes in the open world; while traditional open set recognition methods rely on a large number of training samples, and their performance often drops sharply under small sample conditions. Therefore, the small sample open set recognition problem, which combines the advantages of both, has become a research hotspot in academia and industry.

[0004] The current mainstream small sample open set recognition method is mostly represented by Prototypical Networks (Prototype Network), which calculates the distance between the prototype vector of each category and the query sample to complete classification, and introduces pseudo-open set samples as auxiliary categories to improve the model's ability to recognize unknown categories. This method has enhanced the model's open set recognition ability to some extent, but still has many shortcomings. First, existing methods focus on extracting class features from a single visual modality, failing to fully exploit the potential semantic information, text information or other modal features in the closed set samples, limiting the model's expression ability. Second, the way to build negative prototypes (i.e. prototypes representing unknown categories) is still relatively primitive, often relying on the disturbance of background content or the weighted combination of base class prototypes, lacking systematic modeling and optimization of the negative prototype construction mechanism.

[0005] Although there are studies that try to introduce attention mechanisms to enhance class prototype representation, or simulate unknown class feature space through feature interpolation, the performance of these methods under small sample conditions is still unstable. Especially in the construction of negative prototypes, there are two key problems that have not been effectively solved: one is how to efficiently extract rich information containing semantic, spatial and other multi-modal features from limited samples; the other is how to design an effective mechanism to fuse multi-modal information to generate negative prototypes with strong discriminability for open sets. Limited by the above problems, the current method still lacks the ability to model known classes and unknown classes in small sample scenarios, and new multi-modal guidance mechanisms are needed to make up for this shortcoming. SUMMARY

[0006] Based on the technical problems existing in the background art, the present application proposes a small sample open set recognition method, system and medium based on multi-modal negative prototype, which improves the stability and accuracy of small sample recognition.

[0007] The small sample open set recognition method based on multi-modal negative prototype proposed by the present application is characterized in that a plurality of samples in the query set of the open set small sample task are input into a small sample open set recognition model for closed set classification and open set recognition. The training process of the small sample open set recognition model is as follows: Obtain query set sample images and support set samples and extract support sample visual features, query sample visual features and support semantic features respectively, wherein the support set samples include support set sample images and corresponding support set class names. Input the support sample visual features and the support semantic features into a pre-trained feature fusion network, homogenize the obtained multi-modal features of the class c to obtain a multi-modal class prototype, and the class c is the cth support set class name. Perform attention mechanism operation on the multi-modal class prototype and the multi-modal open weight to output a plurality of multi-modal negative prototypes, calculate the similarity between each query sample visual feature and each multi-modal negative prototype, and take the multi-modal negative prototype with the maximum similarity as the best negative prototype of each query sample. Input the query sample visual features, the multi-modal class prototype and the best negative prototype into an N+1 class classifier to realize closed set classification and open set recognition. Construct a total loss function to adjust the trainable parameters in the small sample open set recognition model.

[0008] Further, in the obtaining query set sample images and support set samples and extracting support sample visual features, query sample visual features and support semantic features, the support set samples include support set sample images and corresponding support set class names, specifically: The support set sample image and the query set sample image are subjected to feature extraction by a visual feature extractor to obtain support sample visual features and query sample visual features, and the support set class name is subjected to semantic extraction by a semantic extractor to obtain support semantic features.

[0009] Further, in the semantic extraction of the support set class name by the semantic extractor to obtain the support semantic features, specifically: The support set class name is input into a word semantic network to obtain short description text information; The support set class name and the short description text information are used to construct an optimized prompt word; The optimized prompt word is input into a pre-trained large language model to obtain class definition description text information; The class definition description text information is input into a pre-trained CLIP text encoder to obtain the support semantic features.

[0010] Further, the query set sample image and the support set sample are obtained as follows: N1 classes are randomly sampled from the base class data as support set classes, and another N2 classes are randomly sampled from the base class data as open set classes, wherein the base class data includes base class images and base class names; For the N1 support set classes, M1 samples are randomly sampled from each support set class to construct a support set sample, and M2 samples are randomly sampled from the remaining samples in each support set class to form a closed set query set; For the N2 open set classes, M3 samples are randomly sampled from each open set class to form an open set query set; The closed set query set and the open set query set are used as the query set sample image.

[0011] Further, the pre-training process of the feature fusion network is as follows: Base class task data including base class images and base class names are obtained; The semantic features of the base class names are extracted by a semantic extractor, and the visual features of the base class images are extracted by a visual feature extractor; The semantic features of the base class names and the visual features of the base class images are input into the feature fusion network to obtain the multi-modal features of the base class images; The average absolute error between the multi-modal features of the base class images and the reference class prototype is used as the reconstruction training loss to optimize the trainable parameters in the feature fusion network.

[0012] Further, in the input of the support sample visual features and the support semantic features into the pre-trained feature fusion network, the obtained multi-modal features are homogenized to obtain a multi-modal class prototype, wherein the multi-modal class prototype The generation formula of the multi-modal class prototype is as follows: ; wherein, is the total number of support set samples of class c, is the index of support set sample corresponding to class c, is a hyper-parameter to control the balance of fusion, denotes a support set sample, is a set of support set samples, is a visual feature extractor, extracts the visual feature of the support set sample, extracts the visual feature of the support set sample, is a semantic extractor, extracts the semantic feature of the support set class name, extracts the semantic feature of the support set class name, is a pre-trained feature fusion network.

[0013] Further, in the output of the multi-modal negative prototype by the multi-layer perception after the attention mechanism operation on the multi-modal class prototype and the multi-modal open weight, specifically: the multi-modal class prototype is taken as a query, and the multi-modal open weight is taken as a key and a value to perform the attention mechanism operation to obtain an initial multi-modal negative prototype of class c; the cross-entropy loss function is used to minimize the similarity between the initial multi-modal negative prototype of class c and the support set sample corresponding to class c, and maximize the similarity with the support set sample corresponding to other classes; the initial multi-modal negative prototype of the support set class is averaged and then passed through Z multi-layer perceptrons to generate Z multi-modal negative prototypes.

[0014] Further, in the construction of the total loss function to adjust the trainable parameters in the small sample open set identification model, specifically: a conjugate training strategy is adopted, and one conjugate task contains two tasks, , , wherein, are two open set small sample identification tasks, are two support set samples, are query samples belonging to the support set class , are query samples belonging to two open set classes; an alignment loss function is used to align the multi-modal negative prototype in the open set small sample identification task representing the same class space with the multi-modal class prototype in the open set small sample identification task ; the total loss function ,in, For open set small sample recognition tasks The loss function is For open set small sample recognition tasks The loss function of .

[0015] A computer system includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program in accordance with the method described above.

[0016] A computer-readable storage medium stores a plurality of classification programs, wherein the plurality of classification programs are used to be called by a processor and execute the method described above.

[0017] The advantages of the small-sample open-set recognition method, system, and medium based on multimodal negative prototypes provided by the present invention are as follows: During the pre-training phase, base class data provides a pre-trained visual feature extractor, feature fusion network, and multimodal open weights for the meta-learning phase, thereby providing stable multimodal expression capabilities for the meta-learning phase. During the meta-learning phase, samples are divided into multiple small-sample tasks, and the multimodal category prototypes are calibrated using multimodal open weights to generate a multimodal negative prototype with generalization capabilities. Finally, based on the similarity between the query sample's visual features and the multimodal negative prototype, the best negative prototype with the strongest adaptability is dynamically selected to complete the N+1 classification task involving known and unknown classes, thereby improving the stability and accuracy of small-sample recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic diagram of the process of the present invention; Figure 2 Schematic diagram of semantic extractor and feature fusion network; Figure 3 Schematic diagram of the multimodal negative prototype generator. DETAILED DESCRIPTION

[0019] The technical solutions of the present invention are described in detail below through specific embodiments. Numerous specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art may make similar modifications without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0020] like Figures 1 to 3 As shown in FIG, the small sample open set recognition method based on multimodal negative prototype proposed in the present invention inputs multiple samples in the query set of the open set small sample task into the small sample open set recognition model to perform closed set classification and open set recognition; The training process of the small sample open set identification model is as follows: Step one, obtain query set sample images and support set samples, and respectively extract support sample visual features, query sample visual features and support semantic features, the support set samples include support set sample images and corresponding support set class names; Step two, input the support sample visual features and the support semantic features into the pre-trained feature fusion network, homogenize the obtained multi-modal features of the class c to obtain the multi-modal class prototype, the class c is the cth support set class name; Step three, after attention mechanism operation on the multi-modal class prototype and the multi-modal open weight, output a plurality of multi-modal negative prototypes through a plurality of multilayer perceptrons; calculate the similarity between each query sample visual feature and each multi-modal negative prototype, and take the multi-modal negative prototype with the maximum similarity as the best negative prototype of each query sample; Step four, input the query sample visual features, the multi-modal class prototype and the best negative prototype into an N+1 class classifier to realize closed set classification and open set identification, where N is the number of support set class names and also the number of multi-modal class prototypes; Step five, construct a total loss function to adjust the trainable parameters in the small sample open set identification model.

[0021] The embodiment includes two key modules: one is a multi-modal class prototype construction module for semantic extraction and feature fusion, and the other is a module for multi-modal negative prototype generation and screening. The method not only extracts image features of the support set and the query set from visual information, but also extracts class semantic information with the help of a large language model and a CLIP text encoder, effectively combines visual and semantic features through a feature fusion network, constructs a multi-modal class prototype, and thus improves the stability and accuracy of small sample identification.

[0022] The embodiment is divided into a pre-training phase of base class data and a meta-learning phase, wherein the base class data provides a pre-trained visual feature extractor, a feature fusion network and a multi-modal open weight for the meta-learning phase, thereby providing stable multi-modal expression capability for the meta-learning phase. In the meta-learning phase, the samples are divided into a plurality of small sample tasks, and the multi-modal class prototype is calibrated through the multi-modal open weight to generate a multi-modal negative prototype with generalization capability. Finally, according to the similarity between the query sample visual features and the multi-modal negative prototype, the best negative prototype with the strongest adaptability is dynamically screened, and an N+1 classification task including known classes and unknown classes is completed.

[0023] In the meta-learning phase of the embodiment, a multi-modal negative prototype generator is trained based on data, as shown in Figure 3 The data construction of the meta-learning phase is as follows: for an open set small sample identification task , it can be defined as: ,in Represents the support set sample set, including closed set categories, that is, support set categories, each category has samples, For category c support set sample images, To support set class names, is the total number of support set samples; For the entire query set, including and Two parts, of which Indicates that it belongs to the closed set category (i.e., the support set category) Query sample image; Indicates that it belongs to the open set category The query sample image, and and The intersection of is an empty set ,Right now, .

[0024] In one embodiment, the pre-training phase of the base task data is specifically as follows: It should be understood that the base class task data and support set samples used in the pre-training stage and the meta-learning stage are randomly extracted from the same base class data. Therefore, the data composition format of the base class task data and the support set samples is similar, and there is a possibility that they have the same data.

[0025] Since the construction of the base class task data is similar to the support set samples in the above meta-learning stage, in order to facilitate the explanation of the pre-training stage of the base class task data, in the following pre-training stage of the base class task data, the above , They represent the base class image and base class name of the base class task data respectively. At the same time, the category c in the meta-learning stage is applied to the category c (the cth base class name) in the pre-training stage. In order to facilitate textual distinction, the old category c is used in the pre-training stage.

[0026] The pre-training phase of the base task data trains a ResNet-12 network (deep convolutional neural network) as a visual feature extractor based on a large amount of base task data. , a feature fusion network and multimodal open weights , specifically including (a1) to (a3): (a1) Obtain base class task data including base class images and base class names through a semantic extractor Extract base class name The semantic features of are the parameters of the semantic extractor; wherein the semantic extractor first queries the base class name by WordNet (Word Semantic Network) , then optimizes the simple definition by a large language model, and finally extracts the visual features of the base class image by a CLIP text encoder : ; wherein, is a CLIP text encoder, is a large language model, is a word semantic network.

[0027] (a2) input the base class image to the visual feature extractor to extract the visual features of the base class image, and the visual features of the base class image and the semantic features of the base class name pre-train a feature fusion network together , wherein and represent the parameters of the visual feature extractor and the feature fusion network respectively, to obtain the multi-modal features of the base class image : ; wherein, is the set of all training samples of the old category c in the base class.

[0028] wherein the visual feature extractor adopts a ResNet-12 network, which is a deep convolutional neural network composed of 12 layers, adopts a residual learning mechanism, and effectively alleviates the gradient vanishing problem in deep networks by introducing a skip connection.

[0029] (a3) finally optimize the feature fusion network by a reconstruction-based training loss, and the reconstruction-based training loss is the mean absolute error loss between the multi-modal features and the reference category prototype : ; ; wherein, is an L1 loss. The reference category prototype is the prototype of the old category c, which is calculated by the average of the visual features of all training samples in the old category c.

[0030] After the pre-training stage of steps (a1) to (a3), a visual feature extractor and a feature fusion network are obtained​ and multi-modal open weight and applied to the meta-learning stage.

[0031] In one embodiment, the meta-learning stage corresponds to steps one to five above, specifically: (b1) obtaining query set sample images and support set samples, the support set samples containing support set sample images and corresponding support set class names; Randomly sampling N1 categories in the base class data as support set categories, and randomly sampling another N2 categories in the base class data as open set categories, the base class data including base class images and base class names; For N1 support set categories, randomly sampling M1 samples from each support set category to construct support set samples, and randomly sampling M2 samples from the remaining samples in each support set category to form a closed set query set; For N2 open set categories, randomly sample M3 samples from each open set category to form an open set query set; The closed set query set and the open set query set are used as query set sample images.

[0032] It should be noted that N1, N2, M1, M2, M3 are integers, and their specific values will be determined according to requirements, preferably M2 is greater than M1, and M2 is equal to M3, for example, as a preferred solution: N1 is equal to 5, N2 is equal to 5, M is equal to any integer from 1 to 5, M2 is equal to 15, M3 is equal to 15, that is, 5 samples are randomly sampled from 5 support set categories to construct support set samples, 5 samples are filtered out from each support set category, and 15 samples are randomly sampled from the remaining samples in each support set category to form a closed set query set; 15 samples are randomly sampled from 5 open set categories to form an open set query set.

[0033] (b2) using the pre-trained visual feature extractor The support set sample images and the query set sample images are respectively subjected to feature extraction, and the support set class names are subjected to semantic extraction using a semantic extractor to obtain support semantic features; Specifically, as shown in Figure 2 The support set class names are input into the short description text information obtained from WordNet; the support set class names and the short description text information are used to construct optimized prompt words; the optimized prompt words are input into the pre-trained large language model to obtain class definition description text information; and the class definition description text information is input into the pre-trained CLIP text encoder to obtain support semantic features.

[0034] It is understandable that the support set class name and short description text are filled into the optimization prompt word construction template, thereby splicing and constructing the optimization prompt word. The optimization prompt word construction process is as follows: Expand {Definition} into a clear, scientifically accurate, single-paragraph definition that explains the core concepts, key features, and application scenarios of {Name}. Use formal academic language to concisely convey its essential meaning. {1. [Ensure Scientific Accuracy]: Based on authoritative knowledge; avoid vague or subjective statements}, {2. [Concise and Focused]: Limit to one paragraph; highlight key points and avoid redundancy}.

[0035] (b3) Input the supporting sample visual features and supporting semantic features into the pre-trained feature fusion network In , we get K multimodal features corresponding to category c, such as Figure 2 As shown; For the visual feature extractors mentioned in (b2) and (b3) and feature fusion network , are obtained in the pre-training stage.

[0036] (b4) Average the K multimodal features corresponding to category c to obtain the multimodal category prototype : ; in, is the total number of support set samples for category c, is the index of the support set sample corresponding to category c, To control the hyperparameters of ensemble balance, Representation category Middle support set samples, is the support set sample set, To use visual feature extractor right The extracted visual features of the supporting samples, To use semantic extractors Support set class name The extracted supporting semantic features, , A set of multimodal category prototypes corresponding to all categories.

[0037] (b5) To model the support space, we first transform the multimodal category prototypes and multimodal open weights Generate an initial multimodal negative prototype for category c through the attention mechanism : ; in, is a learnable parameter, is an activation function, represents transpose, is the dimension of the key vector, which is used to scale the dot product result to avoid the gradient disappearing due to excessive values. Multimodal category prototype and multimodal open weights The attention weight matrix.

[0038] It is understandable that the attention mechanism is based on multimodal category prototypes. As a query, multimodal open weights Scaled dot product attention is performed as key and value.

[0039] (b6) Using the calibration loss function Minimize the initial multimodal negative prototype of category c The similarity with the query set samples corresponding to category c is maximized to optimize the initial multimodal negative prototype;

[0040] in, To query the sample, For query samples Through visual feature extractor The obtained support sample visual features, is the binary cross entropy loss function, is the cos similarity calculation method, For closed query set, the Query sample.

[0041] (b7) Support set category After taking all the initial multimodal negative prototypes, Z multimodal negative prototypes are generated through Z MLPs (multi-layer perceptrons): ; in, For the A multi-layer perceptron, For the Multimodal negative prototypes generated by a multi-layer perceptron.

[0042] (b8) Calculate the similarity between each query sample visual feature and Z multimodal negative prototypes and take the multimodal negative prototype with the maximum similarity as the best negative prototype .

[0043] (b9) Input the query sample visual features, multimodal category prototypes and the best negative prototype into the N+1 classifier to achieve closed-set classification and open-set recognition; Among them, since the support set samples have closed set categories, namely There are support set categories, so N+1 classifiers are set to achieve closed set classification and open set recognition.

[0044] (b10) Adopt conjugate training strategy to optimize small sample open set recognition model. A conjugate task includes and Two tasks, , ,in, They are two open set small sample recognition tasks, are two support set samples, They belong to the support set categories A sample query for are query samples belonging to the two open set categories respectively.

[0045] (b11) Using alignment loss function The number will represent the open set small sample recognition task of the same class space Multimodal negative prototypes and open set small sample recognition tasks Multimodal category prototype alignment; ; in, yes No. Multimodal negative prototypes, for No. Multimodal negative prototypes, yes No. multimodal category prototypes, for No. multimodal category prototypes, It is the cos similarity calculation method.

[0046] (b12) Task The loss function for: ; in, is the calibration loss function, is the alignment loss function, is a hyperparameter, is the cross entropy loss function.

[0047] (b13) Therefore, the total loss function as follows: ; in, For open set small sample recognition tasks The loss function is For open set small sample recognition tasks The loss function of . Consistent, From the open set small sample recognition task The cross entropy loss function, calibration loss function and alignment loss function are composed of.

[0048] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A small sample open set recognition method based on multimodal negative prototypes, characterized by: Input multiple samples from the query set of the open set small sample task into the small sample open set recognition model to perform closed set classification and open set recognition; The training process of the small sample open set recognition model is as follows: Obtaining query set sample images and support set samples and extracting support sample visual features, query sample visual features, and support semantic features respectively, wherein the support set samples include support set sample images and corresponding support set class names; Input the supporting sample visual features and supporting semantic features into the pre-trained feature fusion network, and average the obtained multimodal features of category c to obtain a multimodal category prototype, where category c is the name of the c-th support set class; Perform attention mechanism operations on multimodal category prototypes and multimodal open weights to output multiple multimodal negative prototypes; Calculate the similarity between the visual features of each query sample and each multimodal negative prototype and take the multimodal negative prototype with the maximum similarity as the best negative prototype for each query sample; Input the query sample visual features, multimodal category prototypes and the best negative prototype into the N+1 class classifier to achieve closed set classification and open set recognition, where N is the number of support set class names; Constructing a total loss function to tune the trainable parameters in the few-shot open-set recognition model.

2. The small sample open set identification method according to claim 1, characterized in that: In the step of obtaining the query set sample image and the support set sample and extracting the support sample visual features, the query sample visual features and the support semantic features respectively, the support set sample includes the support set sample image and the corresponding support set class name, specifically: The support set sample images and query set sample images are respectively subjected to feature extraction by a visual feature extractor to obtain support sample visual features and query sample visual features, and the support set class names are subjected to semantic extraction by a semantic extractor to obtain support semantic features.

3. The small sample open set identification method according to claim 2, characterized in that: The semantic features obtained by semantically extracting the support set class name through the semantic extractor are as follows: Input the support set class name into the word semantic network to obtain the short description text information; Use the support set class name and short description text information to build optimization prompt words; Input the optimized prompt words into the pre-trained large language model to obtain category definition description text information; The category definition description text information is input into the pre-trained CLIP text encoder to obtain supporting semantic features.

4. The small sample open set identification method according to claim 1, characterized in that: The acquisition of query set sample images and support set samples is specifically as follows: Randomly sampling N1 categories from the base class data as support set categories, and randomly sampling another N2 categories from the base class data as open set categories, wherein the base class data includes base class images and base class names; For N1 support set categories, randomly sample M1 samples from each support set category to construct the support set samples, and randomly sample M2 samples from the remaining samples in each support set category to form the closed query set; For N2 open set categories, M3 samples are randomly sampled from each open set category to form an open set query set; The closed set query set and the open set query set are used as query set sample images.

5. The small sample open set identification method according to claim 1, characterized in that: The pre-training process of the feature fusion network is as follows: Get base class task data including base class image and base class name; The semantic features of the base class name are extracted by the semantic extractor, and the visual features of the base class image are extracted by the visual feature extractor; The semantic features of the base class name and the visual features of the base class image are input into the feature fusion network to obtain the multimodal features of the base class image; The mean absolute error between the multimodal features of the base class images and the reference class prototypes is used as the reconstruction training loss to optimize the trainable parameters in the feature fusion network.

6. The small sample open set identification method according to claim 1, characterized in that: In the step of inputting the supporting sample visual features and supporting semantic features into the pre-trained feature fusion network and averaging the obtained multimodal features to obtain a multimodal category prototype, the multimodal category prototype The generation formula is as follows: in, is the total number of support set samples for category c, is the index of the support set sample corresponding to category c, To control the hyperparameters of ensemble balance, represents the support set sample, is the support set sample set, To use visual feature extractor right The extracted visual features of the supporting samples, To use semantic extractors Support set class name The extracted supporting semantic features, It is a pre-trained feature fusion network.

7. The small sample open set identification method according to claim 4, characterized in that: After performing attention mechanism operations on the multimodal category prototype and the multimodal open weight, the multimodal negative prototype is output through the multilayer perceptron, specifically: Using the multimodal category prototype as the query and the multimodal open weight as the key and value, an attention mechanism operation is performed to obtain the initial multimodal negative prototype of category c; The cross entropy loss function is used to minimize the similarity between the initial multimodal negative prototype of category c and the corresponding support set samples of category c, and maximize the similarity with the corresponding support set samples of other categories; Support set category After taking the average of all the initial multimodal negative prototypes, Z multimodal negative prototypes are generated through Z multi-layer perceptrons.

8. The small sample open set identification method according to claim 4, characterized in that: In the construction of the total loss function to adjust the trainable parameters in the small sample open set recognition model, specifically: Using the conjugate training strategy, a conjugate task includes and Two tasks, , ,in, They are two open set small sample recognition tasks, are two support set samples, They belong to the support set categories A sample query for are query samples belonging to two open set categories respectively; Using the alignment loss function, the open set small sample recognition task representing the same class space is divided into Multimodal negative prototypes and open set small sample recognition tasks Multimodal category prototype alignment; The total loss function ,in, For open set small sample recognition tasks The loss function is For open set small sample recognition tasks The loss function of .

9. A computer system comprising a memory, a processor, and a computer program stored in the memory, wherein: The processor executes the computer program to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a plurality of classification programs, which are used to be called by a processor and execute the method according to any one of claims 1 to 8.

Images