Small sample open set recognition method and system based on multi-modal negative prototype, and medium

Abstract

The application discloses a small sample open set recognition method and system based on a multi-modal negative prototype, and a medium. The method comprises the following steps: inputting multiple sample images in a query set of an open set small sample task into a small sample open set recognition model to perform closed set classification and open set recognition; a training process of the small sample open set recognition model comprises the following steps: acquiring query set sample images and support set samples, and respectively extracting support sample visual features, query sample visual features and support semantic features; inputting the support sample visual features and the support semantic features into a feature fusion network to obtain multi-modal class prototypes; performing attention mechanism operation on the multi-modal class prototypes and multi-modal open weights to output multiple multi-modal negative prototypes; calculating the similarity between each query sample visual feature and each multi-modal negative prototype, and taking the multi-modal negative prototype with the maximum similarity as the best negative prototype of each query sample; and the small sample open set recognition method improves the stability and accuracy of small sample recognition.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, system and medium for small sample open set recognition based on multimodal negative prototypes. Background Technology

[0002] Few-sample open-set recognition is a research direction that has received widespread attention in the fields of artificial intelligence and pattern recognition in recent years. It aims to enable models to accurately classify closed sets and effectively identify samples from unknown categories, even with only a very small number of known class samples. This technology is particularly suitable for applications with high data acquisition costs, difficult annotation, or complex deployment environments, such as medical diagnosis, autonomous driving, and security monitoring. In these real-world applications, models often cannot predict all possible categories, making the ability to handle unknown categories especially important. For example, in rare disease diagnosis tasks, it is necessary to identify known diseases based on limited samples, and also to have the ability to discover unknown variants or rare diseases to assist doctors in making further judgments.

[0003] Few-shot open-set recognition combines the techniques of few-shot learning and open-set recognition. Few-shot learning focuses on learning new categories with a very small number of samples, emphasizing the model's generalization ability; open-set recognition aims to maintain robustness and recognition ability for categories not present in the training set. However, both have significant limitations: few-shot learning methods typically assume that the test category and the training category come from the same set, making it difficult to handle unknown classes in an open world; while traditional open-set recognition methods rely on a large number of training samples, and their performance often drops sharply under few-shot conditions. Therefore, the problem of few-shot open-set recognition, which combines the advantages of both, has become a research hotspot in academia and industry.

[0004] Current mainstream few-shot open-set recognition methods are mostly represented by Prototypical Networks, which complete classification by constructing prototype vectors for each category and calculating the distance between them and query samples. They also introduce pseudo-open-set samples as auxiliary categories to improve the model's ability to recognize unknown categories. While these methods enhance the model's open-set recognition capabilities to some extent, they still have several shortcomings. First, existing methods mostly focus on extracting category features from a single visual modality, failing to fully explore the potential semantic, textual, or other modal features in closed-set samples, thus limiting the model's expressive power. Second, the methods for constructing negative prototypes (i.e., prototypes representing unknown categories) are still relatively rudimentary, often relying on perturbations of background content or weighted combinations of base class prototypes, lacking systematic modeling and optimization of the negative prototype construction mechanism.

[0005] Although existing research has attempted to enhance category prototype representations by introducing attention mechanisms or simulating the feature space of unknown categories through feature interpolation, these methods remain unstable under small sample conditions. Particularly in negative prototype construction, two key issues remain unresolved: first, how to efficiently extract rich information containing semantic, spatial, and other multimodal features from limited samples; and second, how to design an effective mechanism to fuse multimodal information to generate a negative prototype with strong discriminative power over open sets. Due to these limitations, current methods are still insufficient in modeling known and unknown classes in small sample scenarios, necessitating the development of new multimodal guidance mechanisms to overcome this shortcoming. Summary of the Invention

[0006] Based on the technical problems existing in the background technology, this invention proposes a method, system and medium for small sample open set recognition based on multimodal negative prototypes, which improves the stability and accuracy of small sample recognition.

[0007] The method for identifying small open sets based on multimodal negative prototypes proposed in this invention is characterized by inputting multiple samples from the query set of the small open set task into the small open set identification model to perform closed set classification and open set identification.

[0008] The training process of the small sample open set recognition model is as follows:

[0009] Obtain query set sample images and support set samples, and extract support sample visual features, query sample visual features and support semantic features respectively. The support set samples include support set sample images and corresponding support set class names.

[0010] The visual features and semantic features of the support samples are input into a pre-trained feature fusion network. The multimodal features of the obtained category c are averaged to obtain the multimodal category prototype, where category c is the name of the c-th support set class.

[0011] An attention mechanism is used to calculate the multimodal category prototype and the multimodal open weights to output multiple multimodal negative prototypes; the similarity between the visual features of each query sample and each multimodal negative prototype is calculated, and the multimodal negative prototype with the highest similarity is taken as the best negative prototype for each query sample;

[0012] The visual features of the query sample, the multimodal category prototype, and the best negative prototype are input into the N+1 classifier to achieve closed set classification and open set recognition.

[0013] Construct a total loss function to adjust the trainable parameters in the few-sample open-set recognition model.

[0014] Furthermore, in the step of obtaining query set sample images and support set samples and extracting support sample visual features, query sample visual features, and support semantic features respectively, the support set samples include support set sample images and corresponding support set class names, specifically:

[0015] Visual features of the support set and query set are extracted using a visual feature extractor, and semantic features of the support set class names are extracted using a semantic extractor.

[0016] Furthermore, in obtaining supporting semantic features by semantically extracting the support set class names using a semantic extractor, the specific steps are as follows:

[0017] The support set class name is input into the short descriptive text information obtained from the word semantic network;

[0018] Optimize prompt words by using support set class names and brief descriptive text information;

[0019] The optimized prompt words are input into a pre-trained large language model to obtain category definition description text information;

[0020] The category definition description text information is input into the pre-trained CLIP text encoder to obtain supporting semantic features.

[0021] Furthermore, the acquisition of query set sample images and support set samples specifically includes:

[0022] N1 categories are randomly sampled from the base class data as support set categories, and another N2 categories are randomly sampled from the base class data as open set categories. The base class data includes base class images and base class names.

[0023] For N1 support set categories, M1 samples are randomly sampled from each support set category to construct the support set samples, and M2 samples are randomly sampled from the remaining samples in each support set category to form the closed set query set.

[0024] For N2 open set categories, M3 samples are randomly sampled from each open set category to form an open set query set;

[0025] Closed set query set and open set query set are used as query set sample images.

[0026] Furthermore, the pre-training process of the feature fusion network is as follows:

[0027] Retrieve base class task data, including the base class image and base class name;

[0028] Semantic features of the base class name are extracted using a semantic extractor, and visual features of the base class image are extracted using a visual feature extractor.

[0029] The semantic features of the base class name and the visual features of the base class image are input into the feature fusion network to obtain the multimodal features of the base class image;

[0030] The mean absolute error between the multimodal features of the base class image and the reference class prototype is used as the training loss for reconstruction to optimize the trainable parameters in the feature fusion network.

[0031] Furthermore, in the process of inputting supporting sample visual features and supporting semantic features into a pre-trained feature fusion network, and equalizing the acquired multimodal features to obtain a multimodal category prototype, the multimodal category prototype... The formula for generating it is as follows:

[0032] ;

[0033] in, The total number of support set samples for class c. This is the index of the support set sample corresponding to category c. To control the hyperparameters of the fusion equilibrium, Indicates support set samples, To support the set of sample data, To utilize visual feature extractors right The extracted visual features of the supporting samples To utilize semantic extractors Support set class name The extracted supporting semantic features, This is a pre-trained feature fusion network.

[0034] Furthermore, after performing attention mechanism operations on the multimodal category prototype and multimodal open weights, the multimodal negative prototype is output through a multilayer perceptron, specifically as follows:

[0035] Using the multimodal category prototype as the query and the multimodal open weight as the key and value, an attention mechanism is used to obtain the initial multimodal negative prototype of category c.

[0036] The cross-entropy loss function is used to minimize the similarity between the initial multimodal negative prototype of category c and the corresponding support set samples of category c, and to maximize the similarity between the corresponding support set samples of other categories.

[0037] Support set categories After averaging all the initial multimodal negative prototypes, Z multimodal negative prototypes are generated by Z multilayer perceptrons.

[0038] Furthermore, in constructing the total loss function to adjust the trainable parameters in the small sample open set recognition model, specifically:

[0039] Employing a conjugate training strategy, a conjugate task includes and Two tasks, , ,in, These are two open-set few-sample recognition tasks. These are two support set samples, They belong to the support set categories respectively Query samples, These are query samples belonging to two open set categories, respectively.

[0040] Using the alignment loss function to identify open set few samples representing the same class space Multimodal negative prototype and open set few-shot recognition task Multimodal category prototype alignment;

[0041] The total loss function ,in, For open set few-sample recognition task loss function, For open set few-sample recognition task The loss function.

[0042] A computer system includes a memory, a processor, and a computer program stored in the memory, characterized in that the processor executes the computer program in the manner described above.

[0043] A computer-readable storage medium storing a plurality of classification programs, the plurality of classification programs being invoked by a processor to execute the method described above.

[0044] The advantages of the few-sample open-set recognition method, system, and medium based on multimodal negative prototypes provided by this invention are as follows: In the pre-training stage, the base class data provides pre-trained visual feature extractors, feature fusion networks, and multimodal open weights for the meta-learning stage, thereby providing stable multimodal representation capabilities for the meta-learning stage. In the meta-learning stage, samples are divided into multiple few-sample tasks, and the multimodal category prototypes are calibrated through multimodal open weights to generate multimodal negative prototypes with generalization capabilities. Finally, based on the similarity between the visual features of the query sample and the multimodal negative prototype, the best negative prototype with the strongest fit is dynamically selected to complete the N+1 classification task containing known and unknown classes, thereby improving the stability and accuracy of few-sample recognition. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the process of the present invention;

[0046] Figure 2 This is a schematic diagram of a semantic extractor and a feature fusion network.

[0047] Figure 3 This is a schematic diagram of a multimodal negative prototype generator. Detailed Implementation

[0048] The technical solution of the present invention will now be described in detail through specific embodiments. Many specific details are set forth in the following description to provide a thorough understanding of the invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0049] like Figures 1 to 3 As shown, the small sample open set recognition method based on multimodal negative prototype proposed in this invention inputs multiple samples from the query set of the open set small sample task into the small sample open set recognition model to perform closed set classification and open set recognition.

[0050] The training process of the small sample open set recognition model is as follows:

[0051] Step 1: Obtain the query set sample images and support set samples, and extract the visual features of the support samples, the visual features of the query samples, and the semantic features of the support samples respectively. The support set samples include the support set sample images and the corresponding support set class names.

[0052] Step 2: Input the visual features and semantic features of the support samples into the pre-trained feature fusion network, and average the multimodal features of category c to obtain the multimodal category prototype, where category c is the name of the c-th support set class;

[0053] Step 3: After performing attention mechanism operations on the multimodal category prototypes and multimodal open weights, multiple multimodal negative prototypes are output through multiple multilayer perceptrons; calculate the similarity between the visual features of each query sample and each multimodal negative prototype, and take the multimodal negative prototype with the highest similarity as the best negative prototype for each query sample;

[0054] Step 4: Input the visual features of the query sample, the multimodal category prototype, and the best negative prototype into the N+1 class classifier to achieve closed set classification and open set recognition, where N is the number of support set class names and also the number of multimodal category prototypes.

[0055] Step 5: Construct the total loss function to adjust the trainable parameters in the small sample open set recognition model.

[0056] This embodiment includes two key modules: a multimodal category prototype construction module for semantic extraction and feature fusion, and a module for multimodal negative prototype generation and filtering. This method not only extracts image features from the support set and query set from visual information, but also extracts category semantic information using a large language model and CLIP text encoder. Through a feature fusion network, it effectively combines visual and semantic features to construct multimodal category prototypes, thereby improving the stability and accuracy of small-sample recognition.

[0057] This embodiment is divided into a pre-training stage using base class data and a meta-learning stage. The base class data provides pre-trained visual feature extractors, feature fusion networks, and multimodal open weights for the meta-learning stage, thus providing stable multimodal representation capabilities. In the meta-learning stage, samples are divided into multiple mini-tasks, and the multimodal category prototypes are calibrated using multimodal open weights to generate multimodal negative prototypes with generalization capabilities. Finally, based on the similarity between the visual features of the query sample and the multimodal negative prototype, the best negative prototype with the strongest fit is dynamically selected, completing the N+1 classification task that includes known and unknown classes.

[0058] In the meta-learning phase of this embodiment, a multimodal negative prototype generator is trained based on the data, such as... Figure 3 As shown, the data construction for the meta-learning stage is as follows: for an open-set few-sample recognition task It can be defined as: ,in This represents the support set of samples, containing There are closed set categories, i.e., support set categories, each category has One sample, For category c, the first One support set of sample images, To support collection class names, To support the total number of samples in the set; For the entire query set, including and Two parts, of which This indicates that it belongs to the closed set category (i.e., the support set category). The query sample image; Indicates belonging to the open set category The query sample image, and and The intersection of the two sets is the empty set. ,Right now, .

[0059] In one embodiment, the pre-training phase of the base class task data specifically includes:

[0060] It is important to understand that the base class task data and support set samples used in the pre-training and meta-learning phases are randomly drawn from the same base class data. Therefore, the data composition format of the base class task data and support set samples is similar, and there is a possibility that they have the same data.

[0061] Since the construction of the base class task data is similar to the support set samples in the meta-learning phase described above, for ease of explanation of the pre-training phase of the base class task data, the above-described approach will be used in the pre-training phase of the base class task data as follows. , These represent the base class image and base class name of the base class task data, respectively. At the same time, the category c from the meta-learning stage is applied to the category c (the c-th base class name) in the pre-training stage. For ease of textual distinction, the old category c is used in the pre-training stage.

[0062] In the pre-training phase using base class task data, a ResNet-12 network (deep convolutional neural network) is trained as a visual feature extractor based on a large amount of base class task data. A feature fusion network and multimodal open weights Specifically, it includes (a1) to (a3):

[0063] (a1) Obtain base class task data including base class image and base class name, and extract it using a semantic extractor. Extract base class name semantic features, These are the parameters for the semantic extractor;

[0064] The semantic extractor first queries the base class name through WordNet (Word Semantic Network). The basic definition is then used, and the basic definition is optimized using a large language model. Finally, the visual features of the base class images are extracted using the CLIP text encoder. :

[0065] ;

[0066] in, For CLIP text encoder, For large language models, For word semantic networks.

[0067] (a2) Base class image Input to visual feature extractor Extract visual features from the base class image, and combine the visual features of the base class image with the semantic features of the base class name. Pre-train a feature fusion network together ,in and Let the parameters of the visual feature extractor and the feature fusion network be represented, respectively, to obtain the base class image. Multimodal features :

[0068] ;

[0069] in, This is the set of all training samples of the old class c in the base class.

[0070] The visual feature extractor uses the ResNet-12 network, which is a deep convolutional neural network consisting of 12 layers. It employs a residual learning mechanism and effectively alleviates the gradient vanishing problem in deep networks by introducing skip connections.

[0071] (a3) Finally, the feature fusion network is optimized using the reconstruction-based training loss, where the reconstruction training loss is multimodal features. and reference category prototype Mean absolute error loss :

[0072] ;

[0073] ;

[0074] in, L1 loss. Refer to the category prototype. As a prototype of the old class c, it is composed of the visual features of all training samples in the old class c. The result was obtained through homogenization calculation.

[0075] After the pre-training phases (a1) to (a3), a visual feature extractor is obtained. A feature fusion network and multimodal open weights And apply it to the meta-learning stage.

[0076] In one embodiment, the meta-learning phase corresponds to steps one through five above, specifically as follows:

[0077] (b1) Obtain the query set sample image and the support set sample, wherein the support set sample contains the support set sample image and the corresponding support set class name;

[0078] N1 categories are randomly sampled from the base class data as support set categories, and another N2 categories are randomly sampled from the base class data as open set categories. The base class data includes base class images and base class names.

[0079] For N1 support set categories, M1 samples are randomly sampled from each support set category to construct the support set samples, and M2 samples are randomly sampled from the remaining samples in each support set category to form the closed set query set.

[0080] For N2 open set categories, M3 samples are randomly sampled from each open set category to form an open set query set;

[0081] Closed set query set and open set query set are used as query set sample images.

[0082] It should be noted that N1, N2, M1, M2, and M3 are integers, and their specific values ​​will be determined according to the requirements. It is preferable to set M2 to be greater than M1 and M2 equal to M3. For example, as a preferred scheme: N1 equals 5, N2 equals 5, M equals any integer from 1 to 5, M2 equals 15, and M3 equals 15. That is, 5 samples are randomly sampled from the 5 support set categories to construct the support set samples. The 5 samples already sampled in each support set category are filtered out, and 15 samples are randomly sampled from the remaining samples in each support set category to form the closed set query set. 15 samples are randomly sampled from each of the 5 open set categories to form the open set query set.

[0083] (b2) Using a pre-trained visual feature extractor Feature extraction is performed on the support set sample images and query set sample images respectively. The semantic extractor is used to extract the semantics of the support set class names to obtain support semantic features.

[0084] Specifically: such as Figure 2 As shown, the support set class name is input into the short descriptive text information obtained from WordNet; optimized prompt words are constructed using the support set class name and the short descriptive text information; the optimized prompt words are input into a pre-trained large language model to obtain category definition descriptive text information; the category definition descriptive text information is input into a pre-trained CLIP text encoder to obtain supporting semantic features.

[0085] Understandably, the support set class name and a brief description text are entered into the optimization suggestion word construction template to construct the optimization suggestion words. The optimization suggestion word construction process is as follows:

[0086] Expand {Definition} into a clear, scientifically accurate single-paragraph definition, elaborating on the core concepts, key characteristics, and application scenarios of {Name}. Use formal academic language to concisely convey its essential meaning. {1. [Ensure scientific accuracy]: Based on authoritative knowledge systems; avoid vague or subjective expressions}, {2. [Concise and focused]: Limited to one paragraph; highlight key points and avoid redundancy}.

[0087] (b3) Input the visual features and semantic features of the supporting samples into the pre-trained feature fusion network. In the process, K multimodal features corresponding to category c are obtained, such as Figure 2 As shown;

[0088] For the visual feature extractors mentioned in (b2) and (b3) Feature fusion network All of these were obtained during the pre-training phase.

[0089] (b4) Average the K multimodal features corresponding to category c to obtain the multimodal category prototype. :

[0090] ;

[0091] in, The total number of support set samples for class c. This is the index of the support set sample corresponding to category c. To control the hyperparameters of the fusion equilibrium, Indicate category The Middle Support set samples, To support the set of sample data, To utilize visual feature extractors right The extracted visual features of the supporting samples To utilize semantic extractors Support set class name The extracted supporting semantic features, , This is the set of multimodal category prototypes corresponding to all categories.

[0092] (b5) In order to model the support set space, the multimodal category prototype is first... and multimodal open weights An initial multimodal negative prototype is generated for category c using an attention mechanism. :

[0093] ;

[0094] in, These are learnable parameters. It is an activation function. Indicates transpose. This is the dimension of the key vector, used to scale the dot product result and prevent the gradient from vanishing due to excessively large values. For multimodal category prototypes and multimodal open weights Attention weight matrix.

[0095] It is understandable that the attention mechanism uses a multimodal category prototype. As a query, multimodal open weights Scaling dot product attention processing is performed as key and value.

[0096] (b6) Using the calibration loss function Minimize the initial multimodal negative prototype of class c The initial multimodal negative prototype is optimized by maximizing the similarity between the query set samples corresponding to category c and the query set samples corresponding to other categories.

[0097]

[0098] in, To query samples, For query samples Visual feature extractor The obtained supporting sample visual features The binary cross-entropy loss function is... This refers to the cosine similarity calculation method. To query the categories belonging to a closed set Query samples.

[0099] (b7) Support set categories All initial multimodal negative prototypes are then used to generate Z multimodal negative prototypes through Z MLPs (Multilayer Perceptrons):

[0100] ;

[0101] in, For the first A multilayer sensor, For the first A multimodal negative prototype generated by a multilayer perceptron.

[0102] (b8) Calculate the similarity between the visual features of each query sample and the Z multimodal negative prototypes, and take the multimodal negative prototype with the highest similarity as the best negative prototype. .

[0103] (b9) Input the visual features of the query sample, the multimodal category prototype and the best negative prototype into the N+1 class classifier to achieve closed set classification and open set recognition;

[0104] Among them, because the support set samples have A closed set category, namely There are 10 support set categories, so an N+1 classifier is set up to achieve closed set classification and open set recognition.

[0105] (b10) The conjugate training strategy is used to optimize the small sample open set recognition model. A conjugate task includes and Two tasks, , ,in, These are two open-set few-sample recognition tasks. These are two support set samples, They belong to the support set categories respectively Query samples, These are query samples belonging to two open set categories, respectively.

[0106] (b11) Using the alignment loss function The numbers will represent open set few-sample recognition tasks in the same class space. Multimodal negative prototype and open set few-shot recognition task Multimodal category prototype alignment;

[0107] ;

[0108] in, yes The A multimodal negative prototype for The A multimodal negative prototype yes The A multimodal category prototype, for The A multimodal category prototype, This is a method for calculating cosine similarity.

[0109] (b12) Task loss function for:

[0110] ;

[0111] in, To calibrate the loss function, To align the loss function, For hyperparameters, This is the cross-entropy loss function.

[0112] (b13) Therefore, the total loss function as follows:

[0113] ;

[0114] in, For open set few-sample recognition task loss function, For open set few-sample recognition task The loss function. And Consistent Open set few sample recognition task It consists of the cross-entropy loss function, the calibration loss function, and the alignment loss function.

[0115] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A few-sample open set recognition method based on multimodal negative prototypes, characterized in that, Multiple samples from the query set of the open set small sample task are input into the small set open set recognition model to perform closed set classification and open set recognition. The training process of the small sample open set recognition model is as follows: Obtain query set sample images and support set samples, and extract support sample visual features, query sample visual features and support semantic features respectively. The support set samples include support set sample images and corresponding support set class names. The visual features and semantic features of the support samples are input into a pre-trained feature fusion network. The multimodal features of the obtained category c are averaged to obtain the multimodal category prototype, where category c is the name of the c-th support set class. An attention mechanism is used to compute multiple multimodal negative prototypes from the multimodal category prototypes and multimodal open weights. Calculate the similarity between the visual features of each query sample and each multimodal negative prototype, and take the multimodal negative prototype with the highest similarity as the best negative prototype for each query sample; The visual features of the query sample, the multimodal category prototype, and the best negative prototype are input into an N+1 class classifier to achieve closed set classification and open set recognition, where N is the number of support set class names; Construct a total loss function to adjust the trainable parameters in the few-sample open-set recognition model.

2. The method for identifying small sample open sets according to claim 1, characterized in that, In the process of acquiring query set sample images and support set samples and extracting support sample visual features, query sample visual features, and support semantic features respectively, the support set samples include support set sample images and corresponding support set class names, specifically: Visual features of the support set and query set are extracted using a visual feature extractor, and semantic features of the support set class names are extracted using a semantic extractor.

3. The method for identifying small sample open sets according to claim 2, characterized in that, In obtaining supporting semantic features by semantically extracting the supporting set class names using a semantic extractor, the specific steps are as follows: The support set class name is input into the short descriptive text information obtained from the word semantic network; Optimize prompt words by using support set class names and brief descriptive text information; The optimized prompt words are input into a pre-trained large language model to obtain category definition description text information; The category definition description text information is input into the pre-trained CLIP text encoder to obtain supporting semantic features.

4. The method for identifying small sample open sets according to claim 1, characterized in that, The specific steps for obtaining the query set sample images and support set samples are as follows: N1 categories are randomly sampled from the base class data as support set categories, and another N2 categories are randomly sampled from the base class data as open set categories. The base class data includes base class images and base class names. For N1 support set categories, M1 samples are randomly sampled from each support set category to construct the support set samples, and M2 samples are randomly sampled from the remaining samples in each support set category to form the closed set query set. For N2 open set categories, M3 samples are randomly sampled from each open set category to form an open set query set; Closed set query set and open set query set are used as query set sample images.

5. The method for identifying small sample open sets according to claim 1, characterized in that, The pre-training process of the feature fusion network is as follows: Retrieve base class task data, including the base class image and base class name; Semantic features of the base class name are extracted using a semantic extractor, and visual features of the base class image are extracted using a visual feature extractor. The semantic features of the base class name and the visual features of the base class image are input into the feature fusion network to obtain the multimodal features of the base class image; The mean absolute error between the multimodal features of the base class image and the reference class prototype is used as the training loss for reconstruction to optimize the trainable parameters in the feature fusion network.

6. The method for identifying small sample open sets according to claim 1, characterized in that, In the process of inputting support sample visual features and support semantic features into a pre-trained feature fusion network, and equalizing the acquired multimodal features to obtain a multimodal category prototype, the multimodal category prototype... The formula for generating it is as follows: in, The total number of support set samples for category c. This is the index of the support set sample corresponding to category c. To control the hyperparameters of the fusion equilibrium, Indicates support set samples, To support the set of sample data, To utilize visual feature extractors right The extracted visual features of the supporting samples To utilize semantic extractors Support set class name The extracted supporting semantic features This is a pre-trained feature fusion network.

7. The method for identifying small sample open sets according to claim 4, characterized in that, After performing attention mechanism operations on the multimodal category prototype and multimodal open weights, the multimodal negative prototype is output through a multilayer perceptron. Specifically: Using the multimodal category prototype as the query and the multimodal open weight as the key and value, an attention mechanism is used to obtain the initial multimodal negative prototype of category c. The cross-entropy loss function is used to minimize the similarity between the initial multimodal negative prototype of category c and the corresponding support set samples of category c, and to maximize the similarity between the corresponding support set samples of other categories. Support set categories After averaging all the initial multimodal negative prototypes, Z multimodal negative prototypes are generated by Z multilayer perceptrons.

8. The method for identifying small sample open sets according to claim 4, characterized in that, In constructing the total loss function to adjust the trainable parameters in the small sample open set recognition model, specifically: Employing a conjugate training strategy, a conjugate task includes and Two tasks, , ,in, These are two open-set few-sample recognition tasks. These are two support set samples, They belong to the support set categories respectively Query samples, These are query samples belonging to two open set categories, respectively. Using the alignment loss function to identify open set few samples representing the same class space Multimodal negative prototype and open set few-shot recognition task Multimodal category prototype alignment; The total loss function ,in, For open set few-sample recognition task loss function, For open set few-sample recognition task The loss function.

9. A computer system comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of classification programs, which are used by a processor to execute the method as described in any one of claims 1-8.

Images